Usability Testing: what watching five people actually tells you

The method's value comes from a single design decision: it observes what people do rather than asking what they think. Someone who fails a task and then rates the interface 4 out of 5 is not lying — people attribute failure to themselves, especially when a researcher is watching — but the rating is not evidence about the interface and the failure is.

This is why a usability test that ends with a satisfaction questionnaire and reports the score as the headline has inverted its own logic. The observation is the data. The questionnaire is context.

The corollary is that the test must involve a real task with a real goal. "Have a look around the checkout and tell me what you think" is not a usability test; it is an interview conducted over a screen, and it will produce opinions about colour.

The claim that five users find 85% of usability problems comes from Nielsen and Landauer's mathematical model of the finding of usability problems, which models problem discovery as an exponential curve governed by p, the probability that any one user encounters any one problem.

The result depends entirely on p. At the value used in the original modelling (around 0.31), five users do reach roughly 85%. Where problems are rarer — a complex product, a heterogeneous user base, several distinct user groups — p is lower and the curve is much flatter. Nielsen's own later treatment, Determining Usability Test Sample Size, is considerably more careful about this than the rule's popular form.

The practical guidance that follows is not "five is enough". It is:

• Test five per distinct user group, not five in total. Three user types means fifteen.
• Run several small rounds rather than one large one. Five, fix, five again finds more than fifteen at once, because the second round tests a changed product.
• Expect a flatter curve for complex tasks. Where each user encounters only a fraction of the problem set, five will not get near 85%.
• Do not use five for quantitative claims. Completion rates and times from five participants have confidence intervals too wide to be worth reporting.

The foundational practical account remains Nielsen's Usability Testing chapter in Usability Engineering.

A subscription service tested its cancellation flow with five participants per user group across two groups: recent subscribers and users of more than two years.

The task was written as a goal — "you have decided you do not want to be charged next month; sort that out" — with no mention of settings, subscriptions or cancellation.

All five recent subscribers completed it, with a median of 90 seconds. Four of five long-term users failed. The reason was specific and would not have appeared in any survey: long-term users went to the billing history page first, because that was where the function had lived before a redesign eighteen months earlier. The current path was two clicks away and none of them found it before giving up.

The severity rating put this at the top: high impact (task failure), high persistence (four of five never recovered), and affecting the group with the most tenure. A satisfaction question would have registered mild frustration; the observation registered a flow that a long-standing customer could not complete.

A second-round test after a redirect from the old location was added showed five of five completing. That is the argument for several small rounds rather than one large one — the second round tested a different product.

• Interface vocabulary in the task. Invalidates the finding you were testing for.
• Five users total across several user groups. The rule is per group.
• Quantifying from five. Completion percentages from a handful of participants imply precision that is not there.
• Leading and rescuing. Helping a stuck participant destroys the observation and is very easy to do without noticing.
• Reporting satisfaction as the headline. The method observes behaviour; the score is the weakest thing it produces.
• No severity rating. An unranked list of forty problems gets triaged by whoever shouts loudest.
• One big round instead of several small ones. Iteration is where the method's value compounds.
• Testing a prototype that cannot fail. A click-through with only the happy path wired produces no findings.

A defensible usability test reports: the tasks verbatim, so a reader can judge whether they leaked; the participant groups and how many per group; whether think-aloud was concurrent or retrospective; every instance of facilitator help; and severity ratings with the criteria used.

The single most useful thing to include is the task wording. A study that does not publish its tasks is asking to be taken on trust about the one thing most likely to have gone wrong.

Usability testing finds problems; it does not tell you whether people want the product. A flow can be perfectly usable and address a job nobody has.

It is also artificial. The participant knows they are observed, has no real stake in the outcome, and is working in a session rather than in their day. Some behaviours — abandonment, procrastination, coming back three days later — cannot be observed in a lab at all.

Small samples mean it does not generalise numerically, and severity ratings are judgements that vary between raters. The method's output is a prioritised list of things to fix, not a measurement of quality.

Heuristic evaluation uses experts against established principles and is cheaper, complementary, and prone to finding problems real users never hit.

Cognitive walkthrough works step by step through a task from a theoretical model of the user, and is useful before a prototype exists.

Think-aloud protocol is the technique most usability tests are built on, and has its own methodological literature.

Tree testing isolates navigation structure from visual design, which is the right tool when the failures are about findability.

The output is session recordings and observer notes, and the analysis is comparing across sessions: which participants hit the same wall, at what point in the task, and what they said while they were there.

That comparison is what produces the severity rating, and it is where testing programmes tend to fall back on whichever session the team happened to watch live. Coding sessions against tasks and problems, then retrieving across participants, makes the frequency and persistence dimensions of severity evidence rather than recollection. Evidano supports usability testing as a named methodology and works over session transcripts and notes; the observation itself is the researcher's job, and writing the tasks well is where the study is won or lost.