Testing tools and Google Analytics have a lot of available data. This can be overwhelming. Therefore, it is no surprise I often get the question of what segments to use when analyzing your A/B tests.
A segment is a subset of your data. For instance, users who visited the cart, users on a mobile device, and users from the US. Using these in your A/B test analysis will give you additional insights and learnings.
Things to consider when analyzing segments
Be smart with your time
Be smart about how many segments you analyze. If you spend less time on the analysis, you have more time for new experiments or research. Your analysis is extremely important, but don’t overdo it.
Start with the important segments
There are a few segments you should always check. These are, of course, the segment stated in your hypothesis and differences per device (if you test on several devices).
Other common segments are traffic sources and new/returning visitors.
Extra segments
Next, if useful, choose the most important 2-3 segments for your website, product, or business. More is not needed (unless you want a lot of insights, but the most important ones will give you the essential insights).
These segments can be related to user location, ad campaign, or customers versus prospects.
Things to keep in mind
When analyzing your segments, there are two things to keep in mind.
First, make sure you have sufficient data in each segment. The statistics still apply, so you need sufficient data consistent with pre-test calculations. If you have an interesting insight for a segment with too little data, use it as an indication and set up a new experiment based on it.
Second, be aware of false positives. If you analyze many segments, there is an increased chance of finding a false positive. This means the test data states the variant is a winner, but there is no difference in reality.
For example, if you apply a significance level of 90% and you analyze ten segments, there is a huge chance you will find a false positive.
When you find a winner for a segment not stated in your initial hypothesis, it is wise to rerun the experiment for only that segment.
How to set up segments in GA4
The theories apply to all data tools. How to set them up, depends on the tool you use. As it is highly advised to analyze your A/B tests in your data tool (and not in your testing tool), here is a brief explanation of setting up segments in GA4. If you require more information, please check this article.
In GA4 you can use any of the following methods.
In your Exploration report, select the segments you need. These can be user-based, session-based, and event-based.
In your Exploration report, right-click a dimension in your data and select include or exclude these users (see screenshot).
Finally, you can add comparisons in the default reports of GA4. This has fewer options than the two methods mentioned above.
In summary, choose what is essential for your website and business, and analyze that. Only if you have a lot of time, conduct extra deep dives. And always make sure you have sufficient data.
