After thoroughly analyzing your A/B test, it is time to learn from it. It is critical to understand that your results and learnings are entirely different things.
The importance of learnings
Learnings are very important. Take time to think about what you learned from your A/B test. You will gather valuable insights into your customer’s behavior and come up with new and good A/B tests.
Let me tell you a story about the importance of this. When I started with a former client, I noticed the team was doing many experiments, especially considering the available resources.
Every experiment was analyzed by the data analyst and documented in Airtable correctly. The results of the experiment were shown to the team and the business. After that, they decided whether to implement the change or not and continued with the following A/B test. They did not spend any time drawing learnings from the A/B test. The win rate at that time (the percentage of A/B test winners) was below 15%.
I immediately gave the team a rule when I started with this client. After every A/B test, discuss the following three questions with the whole team. First, what do the results say about the initial hypothesis of the test? For example, if the test was not a winner, was our hypothesis wrong, or the execution? Second, what can we learn from this, and what does this say about the customer’s behavior? And third, what other experiments can we think of now that we know this?
This resulted in almost tripling the win rate!
Questions to ask after each experiment
We all acknowledge that learnings from experiments are just as important as winners and preventing losses.
If you truly want to learn from your A/B tests, answer these questions after analyzing your results.
- What do the results say about the initial hypothesis of the test?
- If it was not a winner: was the hypothesis wrong, or the execution?
- What can you learn when combining these results with what you already know (from previous experiments and research)?
- What could these insights say about your customers’ needs, motivations, and behavior?
- What follow-up experiments can you come up with based on these learnings?
TIP: Place these questions in your templates to keep them top of mind for everyone reporting on A/B tests.
Documentation is key
Documentation is critical for answering these questions. You must know what has been tested on the same page and what (psychological) strategies and customer problems you addressed.
For example, if your test was related to motivation, what did you learn from previous motivation tests on that page. And if you addressed a specific customer problem, what did you learn about that problem in completed experiments on this page and on other pages in the customer journey.
You truly start and keep learning by combining these insights with the results and insights from your A/B test.
Keep learning and build upon your insights
In summary, learnings are critical. Do ask yourself these five questions. It will help you improve your A/B program and build on your learnings and successes. You will gather valuable insights about your customers and, most importantly, get more A/B test winners.
If you want to learn every step in the Conversion Rate Optimization process, do check out my Complete CRO course on Udemy.
