Experimentation

To make the right decisions and create better products, companies need experimentation.

Through experimentation, we want to know what changes to make on the website to increase conversion rates and what products and product features to launch. We also want to improve innovation through continuous experimentation, learn how to serve our customers best, validate every decision, reduce uncertainty in decision-making, and increase alignment with the customer.

As a result, every idea in a company may be heard (tested), we create better products for happier and healthier lives, and we achieve rapid growth.

The difference between CRO and Experimentation culture

To understand the difference between conversion rate optimziation (CRO) and an experimentation culture, we have to understand three things: CRO, culture, and experimentation.

Conversion Rate Optimization

CRO is a way of working. It is a systematic approach to increasing your website’s conversion rates, improving your digital products, and validating your hypotheses and ideas.

CRO mainly focuses on on-site metrics. It generally lives in the marketing department or within a single product or e-commerce team. The goal is to increase conversion rates, and this is very often how organizations start with experimentation.

Culture

Company culture is an organization’s tacit social order. It shapes attitudes, behaviors, and beliefs. Cultural norms define what is encouraged, discouraged, accepted, or rejected.

Company culture is the sum of the formal and informal systems, behaviors, and values built up over the years. At its core, company culture is how things get done around the workplace.

Experimentation

Experimentation is a scientific approach to deciding between two or more competing explanations, decisions, or hypotheses.

These hypotheses suggest reasons to explain something or predict the results of an action.

Ultimately, experimentation is about learning what works best through systematic testing and analysis, leading to continuous improvement and innovation at every level in marketing, product development, innovation and science. It allows us to make data-driven decisions.

Experimentation Culture

Thus, an experimentation culture lives within an organization in which the scientific approach of experimentation is embraced by every employee, from top to bottom.

Unlike CRO, experimentation is not the responsibility of a single person or department. Instead, all employees can define a hypothesis and launch an experiment without permission from management. Experimentation takes place everywhere, not just on the website but also on a strategic and innovation level. Experimentation is completely democratized in an experimentation culture; data trumps opinions, and the organization’s ethos is to think experimentally.

Whereas CRO is a way of working, experimentation is more of a mindset and an entire organization-wide program.

Related articles

Experimentation maturity

Experimentation maturity is the level at which an organization or team has developed its capabilities, processes, and culture to conduct and leverage experimentation in its operations and decision-making effectively. It’s not just about running tests but about how deeply and effectively experimentation is embedded into the organization’s DNA.

Over the years, many experimentation maturity models have been published, each with advantages and limitations. These days, most CRO agencies have their own maturity model, and these are starting to look a lot alike, which is a good thing as we get to a consensus on what experimentation maturity is and the phases leading up to it.

Experimentation Evolution Model by Aleksander Fabijan

An often-used model is one by Aleksander Fabijan from his paper The Evolution of Continuous Experimentation in Software Product Development. The model has four phases of experimentation maturity: crawl, walk, run, and fly. It covers three categories: business evolution, organizational evolution, and technical evolution.

As one of the first models, it helped many organizations assess their experimentation maturity. Also, assigning a maturity phase for each category is possible with several categories presented. For instance, the complexity of the experimentation platform could be in the walk phase, while the experimentation team organization could be in the run phase.

A major downside is that Fabijan wrote this article based on mature organizations. Already in the crawl phase, he talks about a fully centralized data science team. Most companies will simply start with a single CRO specialist. This makes the model unusable for most organizations.

A model that has gained popularity over the last years is the ABCDE model by Thomke, as displayed in his book Experimentation Works.

I love its simplicity. A significant downside, however, is that there is no distinction between different categories. This makes it difficult to see what categories are thriving and which should be improved.

ABCDE model by Thomke
Conversion Ideas maturity model

Most agencies have their maturity model, as do I. My model goes in five steps, from CRO only to an experimentation culture, with the categories of company culture, scope and alignment, team, data, tooling and automation, and process. Within these categories, there are several sub-categories. The model is based on comparing many models, articles, and my years of experience in many companies. You can find the model and full explanation in my Change Management course. You can get the course with a discount on my Online CRO courses page.

My model is a lot more extensive than most models. That is because I made this from a change management perspective. When you write your strategic plan, you want a detailed picture of the current maturity of every team. You want to assess the situation in each (sub-)category per team, as one team might be more mature than others. This way, you can craft a plan for each team to get a culture of experimentation.

Assessing the current maturity

Change model to get to experimentation culture

When your goal is to move towards a culture of experimentation, you first need to assess the current situation. Assessing the current situation is essential to understand how your organization is doing regarding the experimentation maturity right now. With this information, you can craft a change plan to get the organization into the next maturity stage.

Understanding the organization and its goals is your first step to assessing the current situation. Learn about its vision, mission, and long-term strategy. You also want to know about its people, market, growth, target audience, and channels.

Next, to determine the experimentation maturity, you can send a survey to all stakeholders and interview them. You want to know about the company culture, experimentation scope & alignment, team & team goals, data & tooling, and the experimentation process.

Change Management

To successfully move towards an experimentation culture, it is essential to understand that your job is not just to run A/B tests. A large proportion of your work should be dedicated to change management.

Change management is two-fold:

First, you can apply change management tactics on a small scale to create a healthy experimentation environment in your team or department. You can involve people, test their ideas, align your test goals with the organization’s goals, and share useful insights with your colleagues. Most importantly, help your colleagues achieve their goals with experimentation instead of only trying to accomplish your own.

Second, change management is about implementing change, but on a much larger scale. For example, you could restructure departments and work processes and change the hiring process. You need to be a manager or board member for this kind of change.

Company culture

When trying to scale up experimentation, companies often find that the company culture is the biggest obstacle. Shared behaviors, beliefs, and values can make getting to an experimentation culture difficult or impossible.

Common challenges relate to the organization’s mindset, structure, leadership model, and output vs. outcome-driven goals.

Mindset

Three mindsets are essential for an experimentation culture; an experimentation mindset, a learning mindset, and a growth mindset. It means thinking like a scientist with great curiousity, continuously learn and accepting failures, and value change.

Leadership

In the ideal situation, experimentation is fully democratized. This means anyone can set up an experiment without approval from higher management and make decisions based on it. Managers need to set up systems, resources, and standards that allow for large-scale, trustworthy experimentation.

Structure

In the ideal situation, multi-disciplinary teams work together on the same goals, where information and resources are freely shared between the teams. This setup fosters innovation and boosts productivity as skills and knowledge are shared. It also improves decision-making as all teams have the necessary information at hand. However, siloed structures hinder communication and collaboration.

Goals

In an output-driven organization, experimentation might feel like a waste of time, slowing down the whole operation. The most important goal is to ship stuff. If a company aims at outcomes, meaning positive changes for the customers and business, implementing only winning ideas, or non-losing ideas is complete logic. Here, experimentation thrives as it helps achieve business goals.

Experimentation scope & alignment

CRO only

Conversion Rate Optimization is often seen as a tactic performed by a single CRO specialist or team, solely focused on winning tests on the website and thus on the return on investment of experimentation. Tests are generally small front-end changes, and CRO is stuck in the marketing department, e-commerce team, or a single product team. The scope is very limited, making no impact on strategic decisions. It could even happen that the development department does not implement winning tests. This is where many companies start.

Experimentation culture

With an experimentation culture, decisions are validated at all levels of the organization. Experimentation is essential for almost every goal and process in the organization, including strategy and innovation. Experimentation has a significant role in understanding the customer and outcomes, increasing revenues, and mitigating risk. There is a good mix of small, medium, and big disruptive experiments, significantly impacting the most critical business goals and the organization’s direction.

Experimentation team

When organizations start with CRO, one CRO specialist usually does everything. This specialist generally sits in the marketing or e-commerce department. When successful, the team becomes bigger. More specialists are added to the team, and the number of experiments and learnings increases. A developer and UX designer will generally be the first to join the team. Next, an analyst, UX researcher, psychologist, and copywriter could join.

Once management recognizes the value of experimentation for the entire organization and commits to scaling experimentation capability, management can organize its experimentation personnel in one of three ways: A centralized model, a decentralized model, and a center of excellence model.

Centralized model

A team of specialists runs experiments for the entire company. It can focus on long-term projects, improving the company’s experimentation. Projects can be related to building an experimentation tool, developing advanced algorithms, data quality, documentation, and automation. The team also acts as a central point of contact.

The main disadvantage is that business units may have different priorities, which could lead to conflicts over the allocation of resources and costs. They could also resist experimentation, making it impossible for the team to do their work.

The centralized team might feel out of touch with the business units, be less aligned with the goals, and lack specific domain expertise.

Decentralized model

Different CRO specialists or leads operate in different teams. Each specialist is responsible for the experiments that happen in their business unit. There is no centralized team.

The major benefit is that specialists become experts in their domains. Also, they are more aligned with the domain goals and more connected with the team members in their business units.

A major disadvantage is often a lack of information sharing between CRO specialists and no central responsibility for long-term topics such as tooling, documentation, and automation. Besides inefficiency, this also limits learning.

Center of Excellence model

Experienced experimentation specialists operate in a centralized function and others within the different business units. The Center of Excellence model combines the advantages of centralized and decentralized models.

The Center of Excellence supports the specialists in the business units, can focus on long-term projects, and is a central point of contact. It can also promote experimentation and strengthen the experimentation culture throughout the organization by hosting workshops, presentations, and other initiatives.

The specialists in the business units become experts in their domains. They are more aligned with the domain goals and more connected with the team members in their business units.

Data, tooling & automation

It is essential to ensure high-quality data. If no one trusts the data, no one is going to trust your experiments. Do not underestimate this topic, as it is a complete show-stopper for experimentation. Do take care of the quality and educate colleagues to gain trust. The same goes for the quality and reliability of your testing platform and stats engine. Because without trust in data and the tools, growing in experimentation maturity is impossible.

Quantitative data

Google Analytics is likely a tool that immediately comes to mind. But also, Hotjar, Mouseflow, Contentsquare, and Clarity are a few popular tools for visualizing data. As companies grow in maturity, so do the tools. Data collection will improve with server-side tracking, and more data sources will be linked to obtaining a complete cross-device picture of the full customer journey.

U

Qualitative data

For qualitative data, poll and survey tools are popular. Tools like Hotjar and Mouseflow are popular for this. Depending on your product or business, more advanced tools could be added related to usability testing, in-home user testing, shop testing, eye tracking, and neuro-tech. These tools are mostly used for research (to create test ideas and hypotheses), but could also be used for A/B test analyses.

Experimentation tool

Often companies start with a free or affordable experimentation tool, and in the most mature stages, companies have a server-side experimentation platform, either built or bought. The tool has different goals based on your maturity. It starts with simple client-side A/B tests and eventually becomes the essential tool for democratizing experimentation. Everyone should be able to use it easily, safely, and efficiently.

Automation

Automation is crucial to increase the efficiency of experimentation. You can automate recurring work, like funnel analyses, experiment analyses, and statistical calculations. You can also automate safeguards. Automation makes experimentation easier, more accessible, safer, and thus reduces costs.

Experimentation processes

The process can differ greatly throughout the maturity stages. A significant shift occurs when experimentation gets embedded in other processes, like that of the product teams. Where CRO specialists can start with their own CRO process, it needs major adjustments when embedded in other processes.

Conversion Rate Optimization process

The CRO process

The CRO process starts with data, user, and scientific research. Next, test ideas are created and prioritized. For the ideas with the highest priority score, you create a hypothesis, test it, and analyze it.

To increase the number of experiments, you could reduce ideation and design time (i.e., by limiting stakeholder involvement), reduce data analysis time, automate recurring work and learn to code when development is your bottleneck. To increase quality, do proper research.

Double diamond framework

Experimentation in dual track agile

Very popular amongst product teams is the Agile way of working. Agile strives for an iterative approach to project management and software development.

In dual-track agile, product teams become cross-functional and multidisciplinary. Its work is divided into two tracks: discovery and delivery.

In the discovery track, hypotheses are created based on research and validated through experimentation. It focuses on rapid learning and generating validated product ideas, feeding the development backlog.

The Scrum team executes the delivery track. The Scrum team keeps working as it is used to, with the same roles and in sprints. Experimentation can also happen in this track. Sometimes this is automated.

Dual track agile for experimentation

Double/Triple diamond framework

The double diamond framework starts with research to discover customer problems. Next, you define the most critical problem you want to work on. Next is the design phase, in which potential solutions are designed and tested before the final solution is delivered.

The framework is pretty similar to dual-track agile. Here again, an experiment can happen in the design phase, or every final solution can become an experiment.

Occasionally you might also see a triple diamond framework, which starts with the problem discovery and definition. Next is solution discovery and definition. Through research and prioritization, a concept is validated before it gets developed and tested.

Strategic plan

At some point, as CRO or Experimentation lead, you want to write a strategic plan to raise the maturity level of your team or the organization. Writing your plan helps manage the change process. It brings focus to your team’s activities and enables you to track progress toward the goals. Furthermore, it enables you to align with higher management and the board of directors. Without their support, your strategic plan is likely to fail.

The Minto pyramid (displayed in the image below) is a great way to structure your change plan. It was invented by Minto, a McKinsey consultant, and helps you keep your advice concise:

  • Key message: Give your main advice straightaway.
  • Supporting arguments: Support your arguments with high-level insights.
  • Data: Back up your insights with supporting data.

The next step is to present your plan. Here are some tips to make this a success:

  • Get an ally in your audience before you present your plan.
  • Use a deck suitable for presentations.
  • Mention disappointments upfront to manage expectations.
  • Investigate where resistance is coming from if this comes up.
  • You do not need to have a question for every answer. It is okay to acknowledge this and provide the answer after the session.
Minto pyramid

Personal growth

If you have a true optimization mindset, you can apply it to all other areas of life, including your personal growth.

The skills needed in the early stages of experimentation maturity differ vastly from those in mature organizations.

There’s a famous saying in business: What got you here won’t get you there. You will not get the organization to the next maturity stage using the same methods that got it here. You must either learn something new or unlearn something old. Therefore, you must take your personal growth seriously to facilitate change in your organization and grow with the organization and your job.

Receive free CRO tips, the Online Psychology for A/B testing book & more