What segments should you use to analyze your A/B test?

Written by Ruben de Boer

July 5, 2022

Testing tools and Google Analytics have a lot of available data. This can be overwhelming. Therefore, it is no surprise I often get the question of what segments to use when analyzing your A/B tests.

A segment is a subset of your data. For instance, users who visited the cart, users on a mobile device, and users from the US. Using these in your A/B test analysis will give you additional insights and learnings.

Things to consider when analyzing segments

Be smart with your time

Be smart about how many segments you analyze. If you spend less time on the analysis, you have more time for new experiments or research. Your analysis is extremely important, but don’t overdo it.

Start with the important segments

There are a few segments you should always check. These are, of course, the segment stated in your hypothesis and differences per device (if you test on several devices).

Other common segments are traffic sources and new/returning visitors.

Extra segments

Next, if useful, choose the most important 2-3 segments for your website, product, or business. More is not needed (unless you want a lot of insights, but the most important ones will give you the essential insights).

These segments can be related to user location, ad campaign, or customers versus prospects.

Things to keep in mind

When analyzing your segments, there are two things to keep in mind.

First, make sure you have sufficient data in each segment. The statistics still apply, so you need sufficient data consistent with pre-test calculations. If you have an interesting insight for a segment with too little data, use it as an indication and set up a new experiment based on it.

Second, be aware of false positives. If you analyze many segments, there is an increased chance of finding a false positive. This means the test data states the variant is a winner, but there is no difference in reality.

For example, if you apply a significance level of 90% and you analyze ten segments, there is a huge chance you will find a false positive.

When you find a winner for a segment not stated in your initial hypothesis, it is wise to rerun the experiment for only that segment.

How to set up segments in GA4

The theories apply to all data tools. How to set them up, depends on the tool you use. As it is highly advised to analyze your A/B tests in your data tool (and not in your testing tool), here is a brief explanation of setting up segments in GA4. If you require more information, please check this article.

In GA4 you can use any of the following methods.

In your Exploration report, select the segments you need. These can be user-based, session-based, and event-based.

segments in Exploration

In your Exploration report, right-click a dimension in your data and select include or exclude these users (see screenshot).

Include segment from data

Finally, you can add comparisons in the default reports of GA4. This has fewer options than the two methods mentioned above.

Comparison segments

In summary, choose what is essential for your website and business, and analyze that. Only if you have a lot of time, conduct extra deep dives. And always make sure you have sufficient data.

You May Also Like…

Learnings from your A/B tests

Learnings from your A/B tests

After thoroughly analyzing your A/B test, it is time to learn from it. It is critical to understand that your results...

Get the Psychology for A/B testing ebook free

Sign up today to get the book free.

Plus, get free, bite-sized CRO tips and coupons in your inbox.

Please use your personal email address. Business email clients often block automated emails.