What is mostly tested with each method?

In fact, you might have participated in an A/B test, without knowing. Your click or action contributed to a winning version perhaps. However, with user tests… You were invited, you showed up, and were aware of the testing process. 

The A/B tests are more formalized and involve way more users than user tests do.

A/B Test

Call to Actions

The slightest change in a call to action could result in an immediate increase in revenue. CTA’s can be tested for copy, design, location, wording, image, number of words, font size, etc. Even the smallest changes can make a big difference.

Forms

Forms are the first touchpoints where the user is asked to make an important effort, e.g. signing up or buying something. You can test for the number of required fields, field naming, or even if there should be any forms in the first place.

Visual Media

Digital visuals have the power to influence experience and conversions. Researchers test different images to see what sparks more action. They test auto-play vs click-to-play, if users prefer images with or without the people, image size, high-contrast colors, etc.

User testing

New feature

Usability tests allow you to see if the new feature meets the user's expectations. These lead to more refined features that bridge the gap between how you want the feature to be received and how it is actually received.

Usability

Regardless of what you test, each user test provides an insight into the ease of use and user satisfaction. Therefore understanding if users like the product and get around easily is crucial in every user test.

User needs

The needs speak to what your user’s desired outcome is. Usability tests are a perfect way to test if a product has just enough of the right features. The features that satisfy needs specific to the target audience. 

A/B tests are is your go-to tool when you want to compare two versions. With A/B tests you hypothesize, run the experiment, and then view the results in analytics. To run A/B tests you need to know your KPIs (key performance indicators) in advance - otherwise, tests won’t make sense.

User tests, on the other hand, are used to understand if users are able to successfully complete the tasks. The feedback doesn’t come in the form of numbers and stats, but rather in a display of user behavior - usually recorded in a video. 

What is essential to run user tests?

In general, you should run as many tests as often as possible to make sure your product stays on top. 

This is challenging because the more tests you run, the more complicated things get. And with higher difficulty, we tend to overlook the essentials.

Therefore, before engaging in A/B or user tests, make sure you understand the fundamentals.

A/B Test

A/B tests are not effortless. Not even with the help of fully automated testing tools.

First, it takes time to plan and then run the test. Second, it takes knowledge of statistics and sampling to get the right results. Finally, running tests requires some technical steps to deploy the versions and assign those to selected audience segments.

Here are the prerequisites needed to run A/B tests. 

• Hypothesis - it’s not just a random solution to a random problem. The hypothesis needs to consist of the solution, and expected outcome for a specific audience. E.g. If I [APPLY THIS] then [THIS BEHAVIORAL CHANGE] will happen among [THIS GROUP], because of [THIS REASON].
• Audience size  - You should never test on an audience size too small. The tricky part with A/B tests is that you can end up interpreting unreliable results. P.S. don’t forget to use an A/B testing calculator.
• KPI - in order to run the A/B test, you need one single key performance indicator. This KPI should be included within the hypothesis (e.g. if we make a change here, this audience will have a 12% increase in click-through rate).
• Prioritization - The change you want to test should first be evaluated for the potential impact it carries, and the effort needed to test it. Your confidence that the hypothesis will turn out to be true plays a vital role here as well.
• Implementation - A/B tests are run on a live website or app. Accordingly, it’s an option to implement a library or external code in that environment. 

User Test

What you really need is a facilitator and at least two users who can test the product. If you are not sure how to go about running one, then you should get a nice usability test plan on paper, and prepare.

Running usability tests may seem challenging in the beginning, but after one or two, it becomes easier.

Bottom line, go for the A/B tests if you have the needed knowledge and resources. It is costly to assume that green checkmarks on the A/B testing tool are enough. Use the checkmarks as insight and make sure you have all the essentials you need to run tests.

User tests do not require overwhelming prior knowledge, so they are essentially easier to do. In case you want to have your hands on your first usability test, you’ll find this plan valuable.

Three common mistakes made with both test methods

When done right, tests are truly rewarding. 

With the rise in complexity, though, and more people running the tests, opportunities for mistakes have broadened. Here are some issues that can sabotage your returns on testing.

Top 3 mistakes with A/B Tests

Mistake #1 - You don't have tracking set up

Most tests fail before they even start. This is due to the poor analytics account setup. Often, Google Analytics is not chewing the data the right way which leads to wrong assumptions. When running a test anyone can forget to consider different audience segments and traffic sources that make an impact on results. 

Mistake #2 - You finish tests too early 

If there were any rule of thumb in A/B testing, it would be to never stop testing a day in and to aim for at least 95% of statistical significance. The point of A/B testing is to find the truth, not to rule out the alternative you liked less. 

Mistake #3 - You test too many versions

If you are new to A/B testing, don’t test multiple versions all at once. Keep it simple. The more versions you test, the longer it takes. Confirm if the audience is big enough to avoid major traffic overlaps between tests.

Top 3 mistakes with User Tests

Mistake #1 - You test to confirm the bias

Bias is the biggest enemy of all tests. Avoiding or at least being conscious of the bias will spare you from unreliable results. It might also help you avoid making other costly mistakes, such as leading the witness or asking wrong or no questions.

Mistake #2 - You recruit the wrong participants

It happens because teams fail to understand what their average user looks like. What their interests, profession, and hobbies are, or whether they like biking or reading books in their spare time. The formula to nail recruiting is to simply invite people who mirror your average user.

Mistake #3 - You don't give think-aloud reminders

To make a study more objective, it is important to ask the participants to voice their thoughts during tasks. Not reminding them to speak up might lead you to make wrong conclusions about difficulties or their impression of the interface. 

And by the way, silence is one of the main reasons why some researchers are still afraid to go unmoderated. That is why we've built Anna. The most human virtual avatar we could come up with reminds users to think aloud.

Instead of going all-in with either test method, it’s best to start simple. Estimate where you are at currently, what your knowledge level is and what your needs are. From there, make an informed decision. 

If you’re just starting out, do one test only. See how it goes. Check if it went how you expected, and move on to the next test. 

Best tools 

There are many tools to choose from on the market. Here are the ones we think could be most useful to you.

Tools for A/B Tests

Optimizely is an industry-standard enterprise solution for teams. It’s a scalable and reliable conversion optimization tool that allows you to run tests without any need to write code. 

Here is what stands out with Optimizely:

• It’s easy to use - the visual editor and friendly interface are there to help you navigate
• It has advanced targeting and segmentation options
• It’s a full-stack solution - there is more than just A/B tests

VWO works a lot like Optimizely. It includes a rich suite of tools that can scale and help you regardless of the proportions of your team. It’s a very friendly option that more often appeals to individuals and small teams.

VWO is known for its:

• Advanced targeting and segmentation options (behavioral segmentation)
• Code-free approach to building variations
• Reduced load times during the ongoing tests (asynchronous code)

Google Optimize - is a free tool that will cover almost all of your needs. It’s also compatible with Google Analytics. The downside to using this one is that it takes time to familiarize yourself with the platform. In terms of possibilities, there really isn’t a lot you can’t do:

• You can run multivariate tests (run A/B/C/D tests)
• You can use it alongside Google Analytics

Tools for User Tests

We find UserTesting to be a nice fit for larger teams – especially in the US. The tool is reportedly useful across not just research teams, but marketing, product, and development teams. The tool is easy to use and mostly praised for the following features:

• Recruitment from user pools
• Video recording features
• Extensive reporting data

Maze is a very intuitive tool for beginners. Their wide set of user testing methods makes it easy for users to try out different options. Maze is mostly preferred by users due to:

• Easy setup
• Integration with Figma
• Wide range of survey templates
• Real-time metrics

Solid is our freshly crafted automated user testing tool. Solid is a perfect fit for new and experienced researchers who need a reliable and easy-to-use tool. The tool allows users to easily set up and moderate tests with the help of a friendly virtual guide. Solid is the right tool for you if you are looking for:

• Automated test moderation (Anna)
• Reliable software with no setup hassle
• A tool that works with websites, apps, and prototypes
• Rich collaboration built-in features

While we list the ones we deem champions, none of the tools above are able to do the thinking for you. Don’t fall into the trap of thinking 5-star tools handle all the job for you.

It’s probably more helpful to first refresh your knowledge in statistics or testing than to go buy a tool. Expertise is a more solid foundation for successful tests than immediate buying.