Interviews are structured one-on-one question and answer sessions. They are investigative in nature so the intent is to gain understanding in areas that are unclear. The majority of interview questions are “open” and conversational, instead of “closed” and quantitative. For example, an interviewer might ask, “What does our competitor do better than we do?” to initiate a discussion, but not “Please rank on a scale of 1 to 5 which of these competitor’s features are better than ours.”
Interviews are good for learning the customer’s pet peeves, the problems they are running into, their likes and dislikes about the software, and what they want to see in future versions. Interviews cannot determine if software is easy to learn, or easy to use. Interviews will not help identify new market opportunities.
Usability tests are used to evaluate a product design by watching the intended users of the product try it (or a prototype of it) for its proposed use, and seeing what problems are found. Usability tests are good for discovering issues with learning, discoverability, error rates, and speed of use. They also uncover issues with incorrect or omitted feedback. Usability tests can uncover missing features that are needed to complete a workflow. Usability tests cannot discover whether a product will fit into the users’ work environment since they are normally not conducted at the users’ work place, on their hardware and using their files. They cannot verify that a product is solving the right problems for specific users, or if people will actually buy it.
A focus group is a moderated, exploratory discussion with a prepared focus. The participants are users or potential users. Focus groups are good for getting user’s opinions, goals, priorities, and seeing how these compare to others in the group. Focus groups cannot determine if users would actually use proposed new software, or what features should be put into a release. They also cannot be used to determine whether software is learnable or usable.
Surveys are strict sets of questions that are delivered to a large number of people in order to gather quantitative data. Surveys are good for getting simple factual data, priorities, and confirmation of information that was gathered in an exploratory session. Surveys cannot give an understanding of the “why” behind the facts that are gathered since there is no ability to have a discussion with the participants. Surveys also cannot tell if people will buy a product.
Beta tests are when almost-complete software is sent to customers, and they are asked to report problems. Beta tests are good at finding bugs. They are also good at determining if the software actually solves the customer’s problems and if it works in their specific environment. Beta tests cannot convey if the software is easy to learn or easy to use since beta customers will not typically give this feedback. If the software is hard to learn or use, Beta customers will blame themselves and will not report the problem because they do not want to appear stupid.
A demo is a canned demonstration of new software (or new features) shown to your customers to get their opinions. Demos are good for learning what users think about a proposed feature, if a feature would make them more likely to buy, and if they think you are going in the right direction to solve their problems (although they cannot tell you if you are actually going in the right direction). Demos cannot determine if a feature will work in a real production environment, if it is easy to use and learn, or how much people will like the feature after they start using it for real.