What is & What is NOT Usability Testing?

by

Find out what is usability testing and what other user research methods it is often mixed up with.

According to the Harvard Business Review, companies that emphasize human-centered design outperform the S&P 500 by 228% on average.

Usability testing is an invaluable research method for user researchers. If you take the time to learn what usability testing is and how it works, you’ll quickly learn that it can achieve results that no other user research method can.

Most of the world’s biggest companies rely on usability testing in one form or another. Some examples of successful companies conducting regular usability testing include Ryanair, McDonald’s, SoundCloud, and AutoTrader.com. 

In this guide, you’ll find the answers to these questions:

What is usability testing?

Usability testing is a user-centered research technique that involves evaluating user experience (UX) by observing user testers working with the product without them having any prior exposure to it. These users are considered test subjects and during usability tests, they perform usability tasks on products while providing feedback on their experience as first-time users.

Any type of product can be evaluated through usability testing, but the technique is most commonly used for websites and apps. If done right, the insights gained from a usability test can be used in the development process to improve any shortcomings and to refine the user experience of the product before widespread release.

To give an exact usability testing definition, it becomes necessary to break usability testing down into two categories:

  • Formative usability testing is used to research a product that is still in development. Formative usability studies are often repeated during the development cycle and conducted with smaller focus groups.
  • Summative usability testing is used to research a product that is either finished or close to being finished. Summative usability studies are typically conducted on a larger scale and they are used for the final validation of a product prior to its’ release.

A usability test should be treated with the same respect as any other scientific research. That is because usability testing fully relies on high-quality qualitative and quantitative data to be valid. In other words, running a usability test requires significant planning and preparation.

We’ve narrowed the entire process down to seven basic steps in our guide to conducting usability testing. Follow these steps and you’ll be well on your way to solving any usability problems your product or service may have.

According to Carol M. Barnum, author of the bestselling book “Usability Testing Essentials: Ready, set… Test!”, “everyone involved in the creation of websites should be doing their own usability testing”. However, the reality is that very few companies know how to execute usability testing plans effectively. On the other hand, this means that those who are able to do usability testing successfully will be well ahead of the curve.

However, there are many misconceptions about what is and what isn’t usability testing.

What is NOT usability testing?

Usability testing is often confused with other types of UX research methods. In fact, it often appears that there are as many UX research methods as there are UX agencies out there. But, it’s important not to mix usability testing up with other research methods, because it has a very specific and important role within a multi-pronged UX strategy.

The following three types of UX research methods are NOT considered usability testing:

A/B testing

While A/B testing is one of my favorite UX research methods, it’s different from usability testing in that it involves testing different variations of a page on existing users. Usability testing, on the other hand, involves testing on a representative target audience that has never had prior exposure to the product or service under test.

Usability inspections

The difference between usability inspections and usability testing stems from the fact that in the case of usability inspections, the product under test is evaluated by experts. In other words, instead of testing on users, the product undergoes an expert assessment. Usability inspection methods include heuristic evaluations, cognitive walkthroughs, and pluralistic walkthroughs.

Surveys

Although surveys are a valuable tool for collecting remote usability data, they are not considered usability testing. That is because surveys don’t allow user researchers to observe the users testing the product. In addition, survey questions don’t allow for the use of supervised usability task scenarios. This means that the two essential elements of a usability test cannot be used when conducting surveys.

Is usability testing the best UX research method?

Not necessarily.

A/B testing, usability inspections, and surveys can be just as valuable as usability testing under the right circumstances. In fact, they can even outperform usability testing in some cases. 

For example, A/B testing is far superior to usability testing in gathering quantitative data on small design decisions. Let’s examine this by doing a usability evaluation of three call-to-action (CTA) buttons.

Which of these three CTA buttons has the best conversion rate? Interact with each of them to make an educated guess.

I’m using this as an example because I ran this exact A/B test on one of my e-commerce websites. 

The test results demonstrated that the red CTA button on the right had the best conversion rate. It outperformed the green button by 8% and the transparent button by 26%.

Achieving minor data-based insights such as these on the usability of the buttons would have been very difficult to achieve within a usability test. To get enough quantitative usability data for a meaningful conclusion, the test would have required hundreds, if not thousands of test participants.

Conducting a usability test on such a scale would have likely cost tens of thousands of dollars. This A/B test, though did not cost me tens of thousands of dollars. In fact, it cost me virtually nothing – only a single afternoon of setup plus a little waiting for data collection.

A/B testing can be far superior to usability testing under some circumstances. But, the catch is that you need to have an existing user base to A/B test on.

All usability research methods serve a different purpose, have limitations, and have different goals. That is why as a user researcher, you will want to have multiple research methods in your arsenal. This will allow you to maximize the benefits while minimizing the weaknesses of each usability evaluation method.

While a UX research strategy that uses various UX research methods tends to be the most comprehensive and effective, a strategy utilizing only usability testing and nothing else can also bring remarkable results. Start with one method, and grow from there. 

In the long run, though, try not to rely on a single method of user research. As a final goal, try building a comprehensive UX strategy that utilizes the strength and weaknesses of a variety of research methods.

With that being said, let’s cover what’s the exact purpose of usability testing.

What is the purpose of a usability test?

In the most general sense, the purpose of usability testing is to identify usability problems with the goal of improving the user experience of a product. That is done through the observation of representative test users who complete tasks that replicate the behavior patterns of real users.

However, in addition to problem-finding, there are other significant motivators for testing usability:

Collecting qualitative and quantitative data

Turning your intuition-driven design choices into data-driven choices is a powerful change. Using data, you can either validate or invalidate your assumptions, with both of these two outcomes being equally valuable. By collecting qualitative and quantitative data on usability straight from the users, you can skip the guesswork in the design process.

Discovering problems out of the testing scope

It’s often the case that while running a usability test, the test user will steer off the predetermined task scenario and encounter a problem in a feature that was not even being tested. Usability testing allows you to observe the entire customer journey end-to-end and identify any pitfalls on the road.

Saving time and resources

Usability testing is one of the few user research methods that can be conducted with a very small amount of participants. In fact, Jakob Nielsen from NN/g recommends testing with as few as 5 participants for the best results. Conducting a usability test with a tiny sample size like this this requires a relatively small amount of time and money yet it can achieve incredible results.

Why is usability testing important?

In the digital age we live in, products and services with a poor user experience are abandoned in the blink of an eye. Regardless of how big or small the companies behind these products and services are. Any user is always a few clicks away from choosing a competitor.

That is why user experience research is more important now than ever before. Designing human-centered products for your users will enable your business to thrive. On the contrary, though, designing without the end-user in mind will lead to disaster.

When it comes measuring human-centeredness, there’s no better way than using usability testing. Usability testing allows you remove any bias from the equation and gain data-driven insights that can be used to improve the user experience of products.

Understanding what usability testing is, together with its’ benefits and drawbacks enables user researchers to make competent decisions when choosing between various research methodologies.