Usability testing - how many users to test with
A common problem that many teams face when doing usability testing is feedback or sceptacisim from stakeholders about how many people we are testing with. Here is your response.
A common problem that many teams face when doing usability testing is feedback or sceptacisim from stakeholders about how many people the team are testing with - “Five? Shouldn’t we be testing with more?”. This is my response to that challenge.
Over the years I’ve found that explaining your approach upfront before it becomes a problem is most effective. I include it in the initial user research strategy, and talk about it early with stakeholders. I also use it to get commitment from stakeholders (or however is controlling the money) to budget for continual user research/usability testing in small targeted rounds throughout the lifecycle of the service.
How many studies you should conduct and with how many people?
There is no definitive guide as to the exact number of users you should include in your qualitative study. This is different from Quantitative research where you can calculate and reach statistical significance.
The general rule of thumb for qualitative and exploratory research is, the more the better. If you have done zero user research, you have zero validated insights.
A minimum of three to uncover critical issues
In usability studies, the minimum is three for a single study of one user journey from start to finish. From these three you will uncover critical issues.
Target five to validate patterns
By five user studies, you will have more confidence in the identified patterns/insights, and you’ll uncover and collect more unique use cases. Jacob Nielsen wrote a influencial article about this after conducting significant reseach with Tom Landauer, declaring that after testing with five users you will have found 85% of the usability problems. Above this, you would hope to validate the patterns through consistent repetition and find less common (but equally valuable) usability issues.
As you do more repetitive research (same study and focus), the less you will start to learn about the core issues, however, you will uncover new insights that could be less common, but could have a high impact. Plus, you will build data that you can reuse to test against improvements you hope to make in the future.
Continuous iterations - make changes and then test again
It is better to distribute budget for user testing across many small, focused, usability test “rounds” of five users. The goal of usability testing is to improve the design and not just to document its weaknesses – therefore, after creating a new design that aims to address usability issues found in the initial test, you’ll need to test again to ensure the new design actually addresses the usability issues.
Does the target of five user studies apply to all circumstances?
The simple answer is no.
Why? Not all user user journeys will be consistent, they will vary e.g. testing an e-commerce checkout journey with limited paths can be more predictable. Other journeys can be much more complex, involving many more moving parts, actors and cross-cutting processes and systems. In this circumstance the research you conduct will be more exploratory.
The key point here is that the more research you conduct the better.