If you want to achieve more success with CRO, you can apply a simple formula.
But let’s first define what success is. Success with CRO is usually your number of A/B test winners and the number of learnings you got from those experiments. So more winners and more learnings lead to success with CRO.
This leads to the formula: Succes = Chance * Frequency.
The success formula for CRO
Let’s break down the formula:
- Success is the number of A/B test winners and learnings.
- Your chance is the chance of an experiment resulting in a winner. It is all the time you spend on your research, design, coding of your test, analysis of completed experiments, and coordination and meetings with your colleagues.
- The frequency is simply the number of A/B tests you run.
For example, let’s say you spent a lot of time on your optimization research, and by doing so, you increase the chance of finding an A/B test winner to 100 percent. But by spending so much time on your research, you only have time to run one A/B test per month. This will result in 12 A/B test winners per year: 100% * (1 test * 12 months) = 12
If you decrease the time you spend on your optimization research, you decrease the chance of finding a winner.
For this example, let’s assume it drops to 25 percent. But now you have more time to run A/B tests. So the frequency increases to 8 A/B tests per month. This will result in 24 A/B test winners per year, meaning your success with CRO doubles: 25% * (8 tests * 12 months) = 24
This will significantly impact your website and the business in the long run.
Manage your work for more success with CRO
Take a look at this table. If you spend a little less time on your A/B tests, you slightly decrease your chance. For instance, you spend less time on your research, meetings to get approval for an A/B test, making your design less pixel-perfect, and spending less time on analyzing your completed A/B tests. Instead, you spend more time on your frequency.
This will have a huge impact on your success with CRO. It results in a big uplift and much more revenue for your organization in a year.
Of course, when you have difficulty finding A/B test winners, do it the other way around. So, decrease the frequency and spend more time in your optimization research to increase the chance. But in general, for most organizations, I recommend spending less time on your A/B tests, running more A/B tests, and therefore, finding more winners and becoming more successful.
Increase the frequency
When you increase the frequency of your experiments, you increase your success.
To increase your number of A/B tests, you need to reduce your time creating and completing one. There are three ways to do this. Decrease the ideation and design time, learn to code, and decrease analysis time.
1. Decrease the ideation and design time
Let’s start with the first one, the time you spend before setting your A/B test live.
How many hours do you need to create a hypothesis and design? Where can you save time and reduce these hours?
A common problem is that too many people are involved at this stage. Brand managers, product managers, several designers, and other stakeholders have opinions on your A/B test. This costs everyone a lot of valuable time.
As the design is a matter of taste, you will never find the perfect design on which everyone agrees. No matter how many meetings you host and design iterations you have.
The CRO team’s most important goal is that a design matches the hypothesis. Not that it is to everyone’s liking.
To run a large number of experiments, the CRO team should have sufficient autonomy and not be interrupted by too many stakeholders.
If this is a problem within your organization, have an open conversation with your stakeholders. Try to agree with them that a discussion about a test can only arise after it is completed. This will make the conversation much more fact-based as you have the data of your A/B test. Therefore, the discussion will be much more effective, saving everyone valuable time. Time you can spend on creating more A/B tests.
2. Learn to code
Once you have your design, you need to build the test in your testing tool. Very often, the development department is a major bottleneck in this stage. When you want to set up a test, it ends up at the bottom of the IT backlog, costing valuable time and decreasing the number of A/B tests you can run.
To speed things up, you can learn to code. By doing so, you can set up A/B tests with easy and medium complex code while developers build complex experiments. This way, you can triple your number of A/B tests.
At one of my former clients, I set the rule that if a ticket or email to a developer contains more lines of text than lines of code needed to build the experiment, the CRO specialists must build the test themselves. This resulted in a massive increase in the number of A/B tests.
To learn to code, you do need the right course. As a CRO specialist, you don’t build websites. You change them. And this requires an entirely different focus on coding. Therefore, please check out my Coding for AB testing course if you want to learn to code. You will get a discount by using the link on my course page.
3. Decrease analysis time
Finally, to increase your frequency, you can decrease your analysis time.
It is essential to do a thorough data analysis of your tests. But also here, you can save some time. In general, there are three ways to make this happen.
First, decide on the KPI before you run the test. Decide, with your stakeholders, one or two KPIs on which you will assess the test before you start it. This saves time discussing the results afterward.
Second, Set guardrail metrics for your tests. Guardrail metrics are critical metrics for your website, for which you set a threshold before experimenting. When the numbers exceed the threshold, the experiment should be stopped or not declared a winner. You can use these as safety checks.
For example, while the number of transactions is the primary KPI, newsletter subscriptions may not decrease by more than 10%.
And third is, of course, automation. Once you start running many A/B tests, you definitely must look into automating data analysis. It will save you a lot of time and make things much more convenient.
Increase your chance
The most effective way to increase your chance is the time you spend on this side of the equation. There are three ways to increase your chance. The more time you spend on it, the higher your chance of finding an A/B test winner.
1. Spend more time on research & post-test analysis
The first one is apparent. You can increase your chance by spending more time on your optimization research and post-test analysis. So spend more time on user research, data research, scientific research, and insights from your completed experiments. This will help you get better insights and, therefore, better test ideas, resulting in a higher chance.
2. Use an evidence-based prioritization framework
The second method is to use an evidence-based prioritization framework. If your test ideas are prioritized based on the success of previous experiments, you will have a much higher chance of finding an A/B test winner. If you are unfamiliar with evidence-based prioritization, check out my LinkedIn article: https://www.linkedin.com/pulse/5-steps-create-evidence-based-automated-model-feedback-ruben-de-boer/.
3. Build a knowledge database
And the third way to increase your chance is to build a knowledge database.
Building a knowledge database really helps you to achieve more success with CRO. You are building an excellent knowledge database if you keep doing your optimization research and document all your findings, insights, and learnings from your A/B tests.
This database will teach you more about your customers and their behavior, needs, and motivations. And this is very beneficial information as it will help you get a much higher percentage of A/B test winners.
You can share this knowledge with other departments like marketing, product innovation, and product owners.
Besides that, knowing so much about your customers can give you a competitive advantage because you know something about your customer’s behavior that your competitor probably doesn’t know. And you can strengthen this effect. You can become even more knowledgeable when you learn from other departments. Share learnings with the email marketing team, the online marketing team, the offline marketing team, and the data scientists. By sharing learnings, you can strengthen the effect and become more knowledgeable.
So always make sure your documentation is complete and use a good tool. I highly recommend Airtable. If you still need to start using Airtable, check out my free course on Udemy.
For more success with CRO, be aware of the success formula
For more success with CRO, start with the success formula and plan your work accordingly: Success = Chance * Frequency.
To increase your frequency, ensure not too many stakeholders are involved in the design phase, learn to code, decide on the main KPIs and guardrail metrics before running your test and automate. This will save you time to run more A/B tests and, thus, increase your frequency and success.
To increase your chance, spend more time on your optimization research and post-test analysis. Use an evidence-based prioritization framework and build a knowledge database.
But remember, as most experiments will have an inconclusive or losing result, running more A/B tests will most impact your success with CRO. Therefore, spend time increasing your chance, but don’t decrease your frequency.
