A Fundamental Mind Shift For Usability Testing
July 18, 2019
A few days ago, I was watching a user experience expert deliver an interesting talk to a conference of UX professionals. (I do this quite a lot, as I’m always looking for new experts for our All You Can Learn UX video library and future events we might host.)
The presenter was doing an excellent job, making some solid points. I found myself nodding in agreement with every great idea I heard.
Suddenly, the presenter felt the need to impress upon the audience the importance of user research. They said, “You don’t need a big project. A small usability test is all you need. Studies have proven that, with only five to eight users, you’ll find 85% of all the problems in your design.”
My immediate thought: Noooooo! Really?!? It’s 2019. We don’t say this anymore. We never should’ve said this.
This idea, that five to eight users will reveal 85% of all usability problems, is an old myth. It’s not true. It’s never been true.
The origins of the five to eight user myth.
In 1990, Bob Virzi published a paper, Streamlining the Design Process: Running Fewer Subjects, suggesting that for many designs, you could get away with running a usability test with 4 or 5 participants. He had shown a handful of studies where he had ten to fifteen users, but most of the problems showed up in the first four or five. He declared that, if you wanted to fix the obvious problems, that’s all you needed.
In 1993, Jakob Nielsen and Thomas Landauer built on Virzi’s suggestion and tried to refine the mathematical curve, in their seminal paper A mathematical model of the finding of usability problems. Their goal was to find a formula that would tell you how many participants you might need to find 85% of your design’s problems.
Nielsen and Landauer analyzed five usability-testing studies and determined the maximum number of users you needed was eight. They didn’t analyze all usability test studies ever done in the world, they just analyzed five.
Yet, Virzi, Nielsen, and Landauer’s ideas for creating a sample-size prediction formula evolved. It became an untrue rule that you only ever needed five to eight users, no matter what the design was, who the users were, or any other factors.
There was no science saying this. But (almost) everybody believed it. The myth became quite popular.
As often happens, many researchers (including me) published papers to dispel this myth. Even Virzi, Nielsen, and Landauer went on to explain that you couldn’t jump from their findings to the claim that you’d find 85% of all problems.
Yet, the myth persisted. And still persists today. (Or at least, until a few days ago.)
Things have changed in the last 29 years.
One of the usability studies that Nielsen and Landauer analyzed was of Microsoft Office, a 3-year-old product with about a million users. Today, a product with a million users would be considered small. Facebook says it has 2.5 billion users interact with its product every month.
Yet, even though the biggest products have grown 2,500 times the size in 1990, the myth hasn’t changed. The myth is still that you’ll only ever need five to eight users.
Now, some folks have morphed the original myth. They say it’s five to eight users per testing iteration.
OK. Let’s say you’re lucky enough to get ten iterations. Five to eight users for every ten iterations is still only fifty to eighty users total. This number won’t make a difference when you have hundreds of millions or billions of users.
A fundamental shift in how we think of user research.
The big problem with the five to eight user myth isn’t its mathematical impossibility. It’s that it misrepresents the true value of user research.
User research, which usability testing is but one technique, isn’t about identifying all the usability problems in a design. Thinking about it that way reduces user research to being an extension of quality assurance—only finding the flaws we can fix.
The real value of user research comes from increasing our understanding of who our users are. The product or service team should better understand our users with every study, every interview, and every interaction they have. They should constantly increase their understanding of what those users need and the contexts the users work within.
Moving from reactive to proactive research.
The best UX leaders understand research this way. They see that usability testing is the least mature UX research method.
Usability testing is, at best, a reactive technique. The developers build something, then we test to see what flaws they built into it. We iterate as they try to remove flaws, but we’re always reacting.
Getting the organization hooked on meeting our users’ needs.
We can say that usability testing, at best, is a gateway drug. But if we focus on only finding problems and not learning more about our users, we’ll never get our teams hooked on the biggest value of user research.
Instead, we’ll lock them into this mindset that user research is just a variation of QA testing. That it’s about finding bugs, not about delivering better-designed products that truly meet users needs, whatever those needs are.
We need to move away from the problem-finding mindset. The teams that know everything possible about the users are the ones that deliver the best-designed products and services. That’s the mindset change we need.
UX Strategy with Jared Spool
This article was originally published in our new UX Strategy with Jared Spool newsletter. If you’re passionate about driving your organization to deliver better-designed products and services, you’ll want to subscribe.
As we raise our organization’s awareness of what well-designed products and services look like, we need to have solid strategies around how we collect the best feedback from our customers and users. In our 2-day, intensive Creating a UX Strategy Playbook workshop, I’ll work with you and your team to identify the best strategies for building a comprehensive user research approach, that shows where your products and services need to go to become (and stay) best-of-class.
We offer this workshop in cities all over the world. Visit our website at Playbook.UIE.com for more details. We can also bring it to your organization. Contact us to learn how.