What is an experimentation culture?

When you are responsible for Conversion Rate Optimization in your organization, it is essential to understand that your job is not just to run A/B tests. A large proportion of your work should be dedicated to change management to get to a culture of experimentation.

If you really want to thrive with your experimentation program, and you want to change the culture of your organization to a culture of experimentation, you will not get there by simply running A/B tests.

Your job is to motivate your colleagues and higher management, and get them on board and involved. You need to change their mindset and way of working so more people start experimenting, and you get more resources. This is a lot of hard work, but you can change the culture step by step.

Of course, we have the big tech companies like LinkedIn, Facebook, Google, and Booking that created a culture of experimentation from the start of their existence. But also companies without digital roots, such as Walmart, BBC, Nike, IBM, and FedEx, managed to get to a culture of experimentation.

But what is an experimentation culture? And how does it differ from Conversion Rate Optimization?

The difference between an Experimentation Culture and CRO

You want to get from Conversion Rate Optimization (CRO) to an Experimentation culture. So what are CRO and an experimentation culture? And what are the differences?

CRO

Conversion rate optimization is a way of working. It is a systematic approach to increasing your website’s conversion rates, improving your digital products, and validating your hypotheses and ideas.

Conversion rate optimization mainly focuses on on-site metrics. It generally lives in the marketing department or within a single product or e-commerce team. In general, the goal of CRO is to increase conversion rates.

Experimentation culture

To understand what an experimentation culture is, we have to break it down into experimentation and culture.

Experimentation is a scientific approach to deciding between two or more competing explanations, decisions, or hypotheses. These hypotheses suggest reasons to explain something or predict the results of an action.

Company culture is the tacit social order of an organization. It shapes attitudes, behaviors, and beliefs. Cultural norms define what is encouraged, discouraged, accepted, or rejected. It is the sum of the formal and informal systems, behaviors, and values built up over the years. At its core, company culture is how things get done around the workplace.

So an experimentation culture lives within an organization in which the scientific approach of experimentation is embraced by every employee, from top to bottom. Unlike CRO, experimentation is not the responsibility of a single person or department. Instead, all employees can define a hypothesis and launch an experiment without permission from management. Experimentation takes place everywhere, not just on the website but also on a strategic and innovation level. Experimentation is completely democratized in an experimentation culture; data trumps opinions and the organization’s ethos is to think experimentally.

Whereas CRO is a way of working, experimentation is more of a mindset and an entire organization-wide program. It is outcome-driven instead of delivery-driven with a mixture of experiment types. The goal is to validate decisions, understand the customer and outcomes, increase revenue and mitigate risk.

Why experimentation?

We know now what experimentation is, but why is it so important? Why are simply data and conversion rate optimization insufficient?

Data suffers from three limitations

Let’s start with data. According to the book Experimentation Works by Harvard professor Stefan Thomke, big data suffers from three limitations. First, the greater the novelty of an innovation, the less reliable data will be available. If data were available, the innovation would not be very novel.

Second, data is often context-dependent. Just think about best practices in A/B testing. Just because it worked for another website, it does not mean it works for your website.

Third, the analysis of data results mostly in insights about correlation, but not causation. For example, if we collect data for monthly ice cream sales and monthly shark attacks in the United States each year, the two variables are highly correlated. But this does not mean that consuming ice cream causes shark attacks. The more likely explanation is that more people consume ice cream and get in the ocean when it’s warmer outside, thus increasing the chance for a shark attack.

So solely relying on data is a bad idea. The solution is, of course, to combine data with experimentation.

Conversion Rate Optimization is seen as a tactic

This does happen in conversion rate optimization. CRO is important as data shows that only approximately 25% of the changes we make on our websites positively impact the most important goals. The other 75% either makes no difference or even hurts our goals. For some companies, even fewer changes make an impact. This means that all these changes barely result in growth.

Even with excellent research and a fantastic product strategy, it is impossible to predict how changes impact customer experience.

By combining research with A/B testing, you will know what changes to implement and which not. So instead of implementing everything, which barely results in growth, you can implement only winning chances.

This, of course, sounds wonderful. However, conversion rate optimization is too often seen as a tactic performed by a single CRO specialist or a single team, solely focused on winning tests on the website. As CRO specialists have the right mindset, this view completely limits the impact we can make. Unfortunately, the name conversion rate optimization also does not help change that perspective.

We need an experimentation culture to validate decisions

Therefore, we need experimentation. We need an experimentation culture to validate decisions at all levels of the organization. We truly want to understand the customer and outcomes, increase revenues and mitigate risk throughout the whole company.

We want to know what changes to make on the website to increase conversion rates and what products and product features to launch. We also want to improve innovation through continuous experimentation, learn how to serve our customers best, validate every decision, reduce uncertainty in decision-making, and increase alignment with the customer. This results in a massive competitive advantage and realization of rapid growth.

Companies that fully adapted an experimentation culture see this reflected in their stock prices. This image shows the stock performance of companies with a leading experimentation culture compared to the other S&P500 companies. If those other companies do not adopt a culture of experimentation, they might soon perish.

Hence, experimentation is of great importance for all organizations.

Changing the culture

Changing a culture is pretty difficult. As companies try to scale up their experimentation capacity, they often find that the obstacles are not tools, technology, and knowledge but shared beliefs, behaviors, and values. Furthermore, managers have to embrace a new model of leadership, and the organization should accept that not every experiment results in a winner.

To get to a culture of experimentation, you should first be able to assess the current situation. Next, you need to develop a plan for which you need to fully understand what a culture of experimentation entails and all its prerequisites. You need to widen the scope and increase the alignment of experimentation. You also have to know how to set up the best team, set up the right tool stack, and process based on your current maturity level.

Finally, you need to apply change management to convince and motivate higher management and colleagues and implement the changes.

You will learn all of this and more in my Udemy course, ‘Change Management: from CRO to an Experimentation Culture.’