101: Fast UX

1) Agile development is feedback driven.

2) Development teams need to keep testing quick, simple, and frequent.

3) Agile UX must be flexible. It must respond to end user feedback and market changes in one iteration (one week).

That is what ScreenZest provides for them. It is better than other methods of user validation like Interviews, In-house usability tests, Focus groups, Surveys, Beta tests, and Demos. They are just too slow.

It appears what we are selling is “Fast User Validation” –not “Reduced Bounce Rate”.

100: Keep testing quick, simple, and frequent.

Agile development is feedback driven.

Here are some ways salesforce.com facilitates testing:

  • The company uses simple screen-sharing software (GoToMeeting) to share prototypes electronically. Testers pass control of mouse and keyboard to their users to watch them work.
  • A simple conference call lets a facilitator talk to users and lets the user explain likes, dislikes, and areas of confusion.
  • The company sometimes uses TechSmith’s Morae to record the screens and audio as users interact with the software.
  • A projector displays the user’s interactions with the product in a large format so that all the product team members can easily observe them.

A typical weekly testing cycle has salesforce.com designers testing Monday, fixing Tuesday, testing Wednesday, fixing Thursday, and then testing again Friday. By the
end of the week, things are getting pretty good. User-experience people, product man­agers, developers, and others sit in on the call and watch the screen.

99: UX Validation with Interviews, Usability tests, Focus groups, Surveys, Beta tests, Demos

Interviews are structured one-on-one question and answer sessions. They are investigative in nature so the intent is to gain understanding in areas that are unclear. The majority of interview questions are “open” and conversational, instead of “closed” and quantitative. For example, an interviewer might ask, “What does our competitor do better than we do?” to initiate a discussion, but not “Please rank on a scale of 1 to 5 which of these competitor’s features are better than ours.”

Interviews are good for learning the customer’s pet peeves, the problems they are running into, their likes and dislikes about the software, and what they want to see in future versions. Interviews cannot determine if software is easy to learn, or easy to use. Interviews will not help identify new market opportunities.

Usability tests
Usability tests are used to evaluate a product design by watching the intended users of the product try it (or a prototype of it) for its proposed use, and seeing what problems are found. Usability tests are good for discovering issues with learning, discoverability, error rates, and speed of use. They also uncover issues with incorrect or omitted feedback. Usability tests can uncover missing features that are needed to complete a workflow. Usability tests cannot discover whether a product will fit into the users’ work environment since they are normally not conducted at the users’ work place, on their hardware and using their files. They cannot verify that a product is solving the right problems for specific users, or if people will actually buy it.

Focus groups
A focus group is a moderated, exploratory discussion with a prepared focus. The participants are users or potential users. Focus groups are good for getting user’s opinions, goals, priorities, and seeing how these compare to others in the group. Focus groups cannot determine if users would actually use proposed new software, or what features should be put into a release. They also cannot be used to determine whether software is learnable or usable.

Surveys are strict sets of questions that are delivered to a large number of people in order to gather quantitative data. Surveys are good for getting simple factual data, priorities, and confirmation of information that was gathered in an exploratory session. Surveys cannot give an understanding of the “why” behind the facts that are gathered since there is no ability to have a discussion with the participants. Surveys also cannot tell if people will buy a product.

Beta tests
Beta tests are when almost-complete software is sent to customers, and they are asked to report problems. Beta tests are good at finding bugs. They are also good at determining if the software actually solves the customer’s problems and if it works in their specific environment. Beta tests cannot convey if the software is easy to learn or easy to use since beta customers will not typically give this feedback. If the software is hard to learn or use, Beta customers will blame themselves and will not report the problem because they do not want to appear stupid.

A demo is a canned demonstration of new software (or new features) shown to your customers to get their opinions. Demos are good for learning what users think about a proposed feature, if a feature would make them more likely to buy, and if they think you are going in the right direction to solve their problems (although they cannot tell you if you are actually going in the right direction). Demos cannot determine if a feature will work in a real production environment, if it is easy to use and learn, or how much people will like the feature after they start using it for real.

97: Informal UX feedback for Agile Development

Narrowing the gap between gathering usability data and acting on it.

From the software developers’ viewpoint, they can no longer manage the slow pace of formal usability testing. Instead the require faster and more iterative informal testing, by asking for customer feedback on features and also having IT developers in other parts of the company test the product to see if they could break it. It’s a more agile testing approach.

The toolkit has changed to include faster methods. It now takes a shorter time to get a study going, finished, and analyzed.
All this research, modeling, and high level design often occurs in weeks, not months.

Organizations need a continuous flow of users scheduled to validate design work. Schedule users for remote usability testing or site visits well in advance. The one thing you can depend on in an Agile context is that there will always be something.

UX work is mostly about discovery, agile work mostly about delivery. Two worlds – both concerns are necessary. Discovery never stops. And, discovery without delivery isn’t very valuable.

96: Self-initiated UX feedback

UX Metrix is selling an online self-initiated feedback service. Testing produces customer loyalty and reduces support cost.

Common problems with testing:

It’s not uncommon to observe people struggling through a website and afterwards they say it was “easy enough.” Surveys are particularly bad at uncovering “memory” issues, because time passed between the user’s actual experience and the survey.
ScreenZest reminds them of what may have been “right” and “wrong”.

I probably did something wrong.
Many users blame themselves when website problems occur. Issues that make people feel stupid remain under-reported.
ScreenZest helps uncover issues that people would not report by themselves.

Comfort Level
I don’t feel comfortable making suggestions.
People often assume the website they are using is the best that’s possible. Even if a prominent “Contact” option is provided, this may not encourage people to make suggestions.
ScreenZest voting has a lower threshold than originating ideas like with surveys, etc. Voting feedback gives quantitative insights and adding comments gives qualitative insights.

Effort vs. Reward
Should I bother?
Someone who has just spent a lot of time trying to use a site will not be excited to spend even more time trying to provide feedback. People may also wonder whether their feedback will reach the right person and be considered.
Always dignify the customer’s effort with a response (automated or personal), even if you can’t use the suggestion. Follow up for clarification if needed; few people will object to showing interest.

Don’t make me jump through hoops.
Companies frequently create obstacles to reduce contact volume, by burying the contact options many clicks deep, or requiring registration (people hate it), or forcing extensive categorization of the type of feedback, or asking for details that don’t make sense or are privacy sensitive.