The Top 3 Experimentation Mistakes Businesses Make

Effective experimentation enables organisations to optimise their products, services, and processes. But even teams with a solid grasp of A/B testing often stumble down a few blind alleys that undermine their experimentation efforts. These are the three key mistakes we see when people are running experiments. 

Mistake 1: Overcomplicating Experimentation

Experimentation is about learning how you can improve your customer experience, productivity, or processes but sometimes teams inadvertently make it unnecessarily complicated. The overcomplication begins when businesses do manual setups for every experiment, like configuring feature flags or custom metrics for each test. Teams can also overengineer the experimentation process giving them long delays before they see actionable results. We also see businesses at the beginning of their experimentation journey relying too heavily on a single team to create and run experiments, which creates bottlenecks, as every experiment requires their involvement to process results. When any of these things happen, experimentation is slow, cumbersome, and ultimately less appealing to teams.

Complexity also discourages adoption. It should be super simple to set up experiments. If running experiments is too complicated, teams will avoid the process at all costs. After all, if it takes forever to set up experiments, teams are going to see delayed feedback, which will make a huge difference in how they can find creative solutions to potential problems.

How to Overcome It:

Simplicity is key. Experimentation should integrate seamlessly into existing workflows, allowing teams to set up and monitor tests with minimal friction.

Jonas Alves, CEO of ABsmartly, emphasises the importance of making experimentation effortless:

"If experimentation is not easy, if it is not part of the culture, people will try to bypass it completely. It should be super simple to set up a test. Developers and designers should be able to go there and start running experiments without any effort."

The best way to simplify experimentation is to democratise the process. Companies should give all teams the freedom to create and run experiments, give everyone access to a centralised repository for experiment ideas, and establish a lightweight scoring system to evaluate potential impact and feasibility. When businesses standardise how they assess and execute changes, teams can focus on running the most valuable tests without becoming bogged down in unnecessary complexity. Templates can also be a helpful tool that allow you to quickly set up experiments with preconfigured settings, which means the quality across all experiments and feature flags remain consistent. 

Mistake 2: Failing to Build a Strong Experimentation Culture

Leadership needs to support experimentation efforts from the top down, but you also need people to embrace testing from the bottom up—coming up with ideas on how to make experiments better. A business’ experimentation efforts are only ever as effective as the culture supporting them. 

A poor experimentation culture tends to be because there isn’t genuine buy-in at leadership level—and perhaps testing is viewed as a box-ticking exercise rather than a strategic initiative. This mindset then trickles down into teams, leading to siloed efforts where teams don’t share knowledge and their efforts are inadvertently duplicated. Over time, a lack of good quality training for new hires compounds the problem, which further dilutes an organisation’s experimentation best practices.

A fragmented culture leads to an inconsistent approach. Some teams embrace experimentation, while others avoid it. Justifying the resources and focus needed to sustain effective experimentation becomes difficult without leadership buy-in.

How to Overcome It:

Jonas emphasises the importance of constant training and knowledge sharing, based on his experience scaling Booking.com’s experimentation culture:

"At some point, I was training people every week... because every week, we were hiring new developers and designers. That was the biggest effort—to keep the culture intact and help it grow and flourish across the organisation."

To address this, organisations can:

• Train Continuously: Conduct regular team training sessions to ensure everyone understands the why and how of experimentation.

• Centralise Knowledge: Use dashboards and documentation to share learning across the organisation. ABsmartly’s experimentation program dashboard is an excellent example because it enables teams to see how experiments impact business metrics over time.

• Encourage Collaboration: Create a culture where teams share their results, hypotheses, and experiment ideas. This encourages creative thinking and innovation and builds organisational confidence in experimentation.

Mistake 3: Fear of Experiment Interactions

One of the biggest misconceptions about experimentation is the fear of interactions between simultaneous tests. Look at experimentation as the default approach to improve anything in the business because it allows them to make data driven decisions and deliver better results across the board. Although businesses are keen to be able to make faster decisions by trying out ideas and seeing which ideas work they often fixate on whether their experiments will interact. Many teams avoid running parallel tests because they worry that one test might invalidate the results of another. This hesitancy usually comes from a misunderstanding about how interactions occur and the assumption that interactions are more common than they are. 

How to Overcome It:

Here’s what teams should keep in mind:

1. Interactions Are Rare: Most tests do not interfere with each other. In Jonas’ experience at Booking.com, even when running thousands of simultaneous experiments, only a handful of interactions were detected.

2. Intentional Interactions Can Be Valuable: When designed deliberately, interactions can uncover synergies between changes.

3. Monitor But Don’t Panic: Use tools with built-in monitoring for interactions, like ABsmartly’s planned combined report feature, which allows teams to analyse multiple tests as if they were part of a multivariate test.

Jonas advises teams to take a pragmatic approach:

"People are usually so afraid of running many experiments simultaneously, but interactions almost never happen. It’s not a problem to run dozens—even hundreds—of experiments at the same time, as long as you avoid testing on the same element."

Scaling Your Experiments Effectively

You should use experimentation holistically across an entire organisation. Most organisations use experimentation in marketing and product, but you can also run experiments in operations, in customer success, and in your infrastructure. Effective experimentation is all about keeping things super simple, keeping collaboration flourishing. By streamlining processes and democratising experimentation, teams can spend less time worrying about the mechanics and more time on what really matters—learning what works and why.

Ready to unlock growth with experimentation more simply and effectively? Visit ABsmartly for a demo of how our platform can transform the way you test and learn.