Usability Testing Plan

Usability tests are an important part of UX research. Understanding user behavior lays the foundation of a functional UI. In essence, any world-changing app was first a prototype to be tested.

But what is essential for tests to bring in meaningful results? A well-thought-out usability test plan. The right questions, the right format, the right audience, and most importantly, clear goals.

What is a test plan?

Test plans allow designers and researchers to increase the likelihood of finding usability gaps.

A usability test plan is a documented procedure that outlines the information and resources needed for a usability test. It includes test objectives, formats, a list of user tasks and questions, and other test-specific resources, and processes necessary.

The length and complexity of a usability test plan can vary. It can be simple, containing only the most essential data. Given enough importance, it can serve as a central document to keep the project and design teams in the loop.

In this article, we want to keep it straightforward and let you walk away with some practical knowledge.

Now, let’s dive into the most important ingredients your test plan should include.

Define objectives and goals

To ensure you are carrying out an efficient usability test, it’s important to define test objectives upfront. Knowing why you are testing will allow you to design better usability tests.

Your goals don’t have to be complicated. They should just point out what you want to find out from the test. Think of it as directions for usability tests that are clear to you, participants, and your team. Plain and simple.

Your test objectives could be to find out major usability problems, explore user preferences, or to determine how satisfying your user experience is at the moment.

A usability goal can be as simple as "I want my users to sign up without any issues". Or "I want to find shortcuts users take to get to the desired location".

After writing your goals, you’ll be able to drill them down into more specific chunks. For example:

Identify hiccups when a user navigates the new menu
Understand how menu item names are affecting users
Identity what components distract users to sign up

flame

P.S. If you work in a team, make sure you first agree on the goals. Afterward, move on to the next step.

Choose the right method

There are many formats and tools to pick from. Choosing the test setting that fits best will depend on four factors (or lack of them). Your budget, your available time, your expertise level, and the complexity of your test.

Should you go with moderated tests?

Moderated usability tests include sessions in which moderators actively engage users. It's a remote or in-person type of testing where the moderator guides users through the tasks.

Researchers favor moderated tests when:

They are experienced in interviewing
They are able to clarify tasks or ask follow-up questions
They need to guide users through low-fidelity (paper) prototypes or challenging tasks
They want to remind users to think out loud

Moderated tests will naturally leave space for follow-up questions, feedback, and guidance. These tests, while effortful and time-stretching, enable you to gain deep qualitative insights.

If done in person, researchers can even get a grasp of subtle body language signs and reactions.

In case you go for remote moderated tests, make sure you use validated video conferencing or user testing tools. Prepare the tech setup, try out the software first, and expect difficulties.

Finally, try to avoid testing low-fidelity paper prototypes or technically challenging software remotely.

If you are a beginner or need a more efficient way of testing more users, you might first want to consider picking the second alternative - unmoderated usability tests.

When to choose unmoderated tests?

Unmoderated tests are done without human supervision. Test users are given instructions to think out loud and are recorded while they do the tasks. Most unmoderated tests today are done remotely with the help of user testing tools.

Unmoderated tests are a quick and efficient way to run more tests.

There are plenty of convenient user testing tools to choose from. UX researchers go for unmoderated tests when they want to save time, test more, and see tangible results.

You can go for unmoderated test tools if:

You have a fixed set of clear tasks
You don’t need to interrupt and ask questions during tests
You expect testers to be able to complete the tasks
You need to squeeze more tests in less time

The problem with unmoderated tests is that users rush through tests without speaking out loud. If you are running unmoderated tests, it is challenging to prompt users to speak.

A solid amount of test tools fail to remind participants to speak up consistently. A challenge Solid is tackling with the help of a virtual avatar that reminds users to vocalize.

Feel free to check out how our avatar Anna can moderate your user tests.

Select the right test users

Most usability tests are done with 5-10 test participants. Regardless of the format, kicking off with five is a good starting point. With this amount, you increase the chance to discover usability issues 1/3 of your users have.

Plus, with a small sample size, it never gets too complicated to collect insights at the end.

There are two options to choose from when it comes to participant selection. You can recruit your own users or pick participants from existing user pools.

Recruiting your own users can be a straightforward process. We recommend following a “Who to ask first” hierarchy, starting with clients as your number #1 priority.

Ask your clients - they are the ones that will be using your software
Ask your colleagues - convenient, but make sure they haven't seen the software
Ask your peers - they know exactly what you need, but are usually too tech-savvy
Ask your friends or family - they would do anything to help but might be too nice

In contrast, pulling participants from the third-party user pools will enable you to mirror the audience you are designing for. There are many options to choose from, such as TestingTime or User Interviews. They offer a wide variety of features, are easy to use, and allow screener questions.

If you’re starting out, start simple. Try it out, an imperfect test is always better than no test.

Prepare the usability test outline (briefing)

User test duration can range anywhere from 5 to 60 minutes, depending on the number of tasks and format. However, most tests take anywhere from 15 to 30 minutes.

A good rule of thumb is to prepare 2-5 tasks per usability test. The more tasks, the harder it gets for the user to keep focus.

Regardless of what test format you go for, it’s useful to have these written down as part of the usability test plan.

Welcome introduction
Warm-up questions (pre-test questions)
3-5 clear and simple tasks
Questions to ask during the test (in-test questions)
Test wrap-up (post-test questions)

Here a possible outline:

1. Introduction - Hi [Name], my name is Marcel. I have some information for you. I'm going to read it to you, and we'll make sure everything is clear.

We are testing a website to see if it works as intended. You will help us do just that with a couple of guided clicks and tasks. Before we begin, I will help you set everything within your browser to start the testing.

The test will take no longer than 5 minutes, and remember, we are testing a website, not you. There is no way in which you can make mistakes here. We actually like witnessing you having difficulties, because that means we have found the space for improvements.
2. Warm-up questions - How are you feeling today?

What websites or apps do you use to buy online?

On a scale from 1-10 where 1 is ‘awful,’ 5 is ‘neutral,’ and 10 is ‘great’, please rank how you like the Amazon web design.
3. User Tasks - Find one book by J.K. Rowling and put it in a cart to buy later.

Search for skateboards, filter only 5-star items, and put the first one in the cart.

Go to your cart from the navigation menu and purchase two items from the cart.
4. In-task questions - Could you speak aloud and explain your thoughts?

Why do you think the 5-star filter is confusing?
5. Test closure - From 1 to 5 how would you rate the overall experience on the website?

What did you find most frustrating while navigating the website?

What, if anything, surprised you the most?

Prepare the task supporting document

Tasks and questions are an integral part of the usability test. Setting these wrong or putting too many details into one single question can leave users overwhelmed.

Therefore, before giving tasks to users, it’s helpful to first prepare a supporting document. Here, you would write down all task-relevant information, what research question you want to be answered, and the task itself.

It’s the document you prepare for yourself and carry as a cheat sheet for tests.

So, make sure you have these items lined up in advance:

Task name - Go for a descriptive name
Research question - Write down the expected takeaway
Task inputs - List task-relevant resources (login, credit card information, etc.)
Task - Specify instructions for the tester
Expected scenario - Predict users’ behavior
Notes - Write down related information
Key findings - Conclude the results

With our example from above, your task supporting document could look like this:

Task name

Find a J.K. Rowling book
Research question

What are the pain points in an eCommerce shopping journey?
Task inputs

Test credit card: 3142 3238 434 9203, John Doe, 12/22, 123
Task

Find the list of all J.K Rowling’s books. Choose the one you like, find reviews & book description, and add it to the cart
Expected scenario

User will start the journey by navigating to the search bar and typing in the author's name. The user will then click on the author profile and find all listed books. Users will not use any filtering options, but rather open 2-3 books to check user reviews and what the book is about. Afterward, they will add one book to the cart.
Notes

Remind users to speak up, check if different age groups react differently
Key finding

78% of users ended up using the search menu, 38% of users immediately went for the search menu, 16% of users went to the fiction category and filtered out top writers, search bar loses the user input after page load

Do a trial test

It’s helpful to first make a trial round before running the test with 'real testers'. A trial test is done on a small-scale sample before rolling it out to more participants. Sometimes, even just one colleague or a friend will help you identify test flaws.

If you are a beginner, trial tests can help you prepare or throw away unnecessary elements. Also, if it’s not your know-how area, tryouts will help you to better prepare for the real usability tests. You can identify errors as well as rephrase tasks and research questions.

Just remember that even experienced researchers are running trial tests. They understand users often behave differently than expected. Trial rounds can give the opportunity to set better expectations for user behavior.

Don’t wait. Start!

The best time is now.

These test planning tips are designed to help you set up your own usability tests. Your planning doesn't have to be super detailed and extensive - if you follow our tips, you'll have your own plan set up in no time.

Written by

Luciano Kovaevi

Content Writer @ Solid