Usability test sample size
“How many users should we include in the usability test?” might be the question many of us have asked at least once; it is such a question that requires consideration of expertise and effort. On one hand, deciding on a way to approach the matter is dependent on one’s capability to handle the ways to do it; on the other, project limitations such as budget and time have to be taken into account as most of us do not have unlimited access to either of these.
A simple answer to the question is, in fact, based on the statistical significance of a test which is the ability to explain the likelihood of an event not happening randomly. So if you set a significance level of 90%, it means that you are 90% confident that the results of the test you conducted can be explained by your hypothesis.
What affects the significance level is the test sample size, confidence level, and hypothesis type as the methods you will need to apply are based on these. When your hypothesis is built on your ability to find pain points of the users during a test, you can find how many users you need to test by deciding on how confident you want to be in the results. This is just a simple explanation of how a sample size can be found before a test is conducted at the most basic level. Since it is a prediction, as you might have noticed, we can refer to the whole process as inferring based on possibilities. Therefore it is to be noted that there are different methods for calculating the sample size depending on the distribution of the probability. Luckily, there are already analyses with actual data on the matter that let us not delve deep into the matter here but explore what kind of approaches are available to us.
Quantitative Usability Tests
An analysis on the matter sets a minimum of 20 users as a requirement for the test sample size if you want to conduct a quantitative study. There are strict conditions, however, regarding how the process of the test should be planned which results in a trade-off between statistical accuracy and cost in terms of effort and money. If you want to take this approach, make sure that you have the expertise and resources to see the test through.
Qualitative Usability Tests
Another analysis suggests that only 5 users are needed for a usability test by conducting a qualitative study. By doing so, the analyst argues that the benefit provided as opposed to the cost would be much higher since:
“You don’t have to measure usability to improve it.”
If you see your users, for example, struggling with navigating back to the page where they sign-up or place an order due to a design element, the matter of how much they have a hard time, loses its importance in terms of ability to capture pain points. If a sample size of 5 users has the probability of detecting the signals, you can allocate the remaining budget conducting more tests.