User Experience GuideResource: Usability Test Quality Heuristics
The following indicators can help determine if a usability test will produce useful results.
This is not an exhaustive list, but it should be helpful in planning and assessing usability tests. Additions are welcome!
Study design
| Indicator | Good sign | Bad sign |
|---|---|---|
| Study purpose | Researcher clearly articulates the purpose of the usability test (for example, as a specific question to answer, an area of inquiry, etc.). | Researcher does not specify a purpose for the test, or the purpose specified is very broad, like “testing the app” or “finding problems.” |
| Number of participants | Tests include a sufficient number of participants necessary to see patterns, and there’s only one participant per session. | Study includes only one participant, or includes more that two participants testing simultaneously (like a focus group). |
| Sampling bias | Participants are a diverse set of people who actually use the application (or do the task the application is supposed to support). | Participants are experts (vs. average users), friends, family, or close colleagues of the product team. |
Moderator style
| Indicator | Good sign | Bad sign |
|---|---|---|
| Question style | Researcher asks follow up questions that are open-ended and non-leading. | Questions are leading or subtly suggest potential solutions or hypotheses. |
| Task variability | Researcher asks participants to complete similar tasks related to the study’s purpose. | Researcher asks participants to complete different tasks, or tasks unrelated to the study’s purpose. |
| Framing | Researcher asks participants to complete tasks with the product or service that align with their work-related goals. | Researcher asks participants to complete tasks unrelated to their work-related goals. For example, asking a participant how they might send a fax when their job doesn’t call for that. |
| Priming | Moderator asks participants to complete tasks without indicating how that might be done. For example, “How would you view the status of your application?” | Researcher guides participants in completing tasks. For example, “Which of the links in the header would you click to login?” |
Team participation
| Indicator | Good sign | Bad sign |
|---|---|---|
| Number of teammates | The team designates a single moderator for the test, and at least one member of the product team observes the usability test. | A single person from the product team participates in and leads the test. |
| Observer role | Observers do not moderate. They are generally quiet, and ask open-ended questions after the test has concluded. | Observers interrupt the participant, or attempt to sell or explain the product. Observers debate the participant’s actions in their presence. |
Sensemaking
| Indicator | Good sign | Bad sign |
|---|---|---|
| Notetaking | Tests are recorded or notes are taken for sharing with absent stakeholders. | Tests are not recorded, or test results are not documented. |
| Synthesis | Moderator debriefs with teammates immediately after each interview. Researcher looks for patterns across multiple participants and surfaces problems that affected several people. | Moderator reports the most memorable problems without conducting affinity mapping or some other analysis technique. |
| Presentation of findings | Researcher reports findings to team and stakeholders in an easy to follow, well prioritized way. | Researcher presents the team a “basket of issues” or an unprioritized laundry list of potential changes. |
| Incorporation of findings | Product team translates findings into future user stories or product refinements. | Researcher reports do not affect the product backlog or ongoing development work. |
