Debunking the Myths: What AB Testing Is (and Isn't)

Online controlled experimentation is a powerful tool that can help a business to make better decisions and drive real growth. But let’s be honest, there’s still a lot of misunderstanding about what experimentation actually is and what it can deliver. Experimentation is a rigorous process designed to test theories and understand cause-and-effect relationships. An experiment typically involves comparing results between a control group and one or more experimentation groups to measure the impact of a specific change. A/B testing is the most straightforward version of an experiment involving just one control group and one experimentation group. 

While experimentation and A/B testing share the same goal of uncovering causal insights, A/B testing is, in effect, a subset of experimentation, which is designed for quick and straightforward comparisons, while experimentation is a broader concept that can have multiple experimentation groups, control groups, and complex designs to answer a wider range of questions. 

Some people—including product teams, designers, developers, or management—believe that experimenting stifles creativity, is too costly, and delays change. But those are myths, and this blog sets out to explain what experimentation is (and isn’t) by busting some myths.

Let’s use three fictional businesses to illustrate how effectively they use experimentation to compare the outcomes of a company with and without A/B testing. 

• RapidChange doesn’t buy into the usefulness of experimentation, so they dive into implementing changes as fast as possible.  

• EasyChange is keen to try experimentation and implement any change that shows a positive outcome after a reasonable sample size is achieved.

• ConsideredChange A/B tests every change, but all decisions are guided by robust statistical practices.

Myth #1: Experimentation is tactical

When people think about A/B testing, they often picture something fairly tactical—swapping out a button colour or testing a new headline. While that’s part of it, experimentation is so much more than a few tweaks here and there. At its core, it’s a strategic approach to decision-making. 

RapidChange, dismisses the value of experimentation altogether, they focus on implementing changes to their app and website as quickly as possible, believing that speed outweighs caution or learning. This leads to inconsistent results—some improvements are effective, but others alienate customers or create unintended, sometimes harmful problems. Without a framework to guide decisions, RapidChange’s growth is erratic and riddled with costly missteps. While EasyChange is enthusiastic about experimentation but often implements changes based on inconclusive evidence, leading to mixed outcomes and missed opportunities for sustainable growth. 

In contrast, ConsideredChange takes a much more strategic approach to decision-making. Every decision they make is backed by a carefully crafted A/B test. For example, before launching a new feature, they test its impact on a small segment of users and evaluate the results with rigorous statistical analysis. This means they only implement the changes they know enhance the user experience, driving steady and sustainable growth.   

Myth #2: Experimentation squashes creativity

There’s a persistent myth that testing stifles creativity as if data-driven decision-making somehow kills innovation. Experimentation doesn’t shut down bold ideas—it gives them a chance to shine. Testing gives a business a framework to safely explore creative concepts, ensuring they work for the audience before fully committing to them and democratising experiments so that the whole team can get involved. 

ConsideredChange demonstrates how using an experimentation platform fuels creativity responsibly. Let’s say they’ve developed a bold, minimalist homepage redesign. Rather than taking a gamble by rolling it out to all users, they test it on a small subset. The data shows the redesign improves engagement for one user segment but deters another. Armed with this insight, they refined the design to balance both groups' needs, creating an innovative solution grounded in evidence.  

Myth #3: Experimentation is a one-time thing

Many businesses make the mistake of treating experimentation as a one-and-done activity. They’ll run one A/B test, get a result, and move on, assuming they’ve ticked the “experimentation” box. But experimentation isn’t a one-off project—it’s an ongoing learning and discovery process.  

By skipping experimentation entirely, RapidChange misses out on the opportunity to learn from their mistakes. Each rushed change might create short-term wins but fails to build a cohesive strategy for long-term growth. And who knows what they might break along the way. 

Companies like ConsideredChange treat experimentation as a continuous journey. Each test builds on previous findings, contributing to a growing knowledge base about their customers. Over time, this iterative approach helps them refine their offerings, improve their decision-making, and achieve consistent growth. Experimentation is about cumulative insights, mindset and velocity rather than one-time, quick fixes, and ConsideredChange shows how patience and persistence pay off.  

Myth #4: Experimentation is just guesswork

Effective experimentation is rooted in science, not chance. It relies on clear hypotheses, robust statistical methods, and careful analysis to ensure trustworthy and actionable results.  

Consider the contrast between EasyChange and ConsideredChange. EasyChange runs tests without stringent statistical thresholds, interpreting results quickly and implementing changes based on what seems to work. While this avoids outright failures, it often leads to decisions based on incomplete or unreliable data.  

ConsideredChange, however, prioritises precision. For every test, they ensure the sample size is sufficient and use rigorous significance thresholds (e.g., p < 0.05) to validate results. By focusing on statistically robust methods, they can confidently implement changes that deliver meaningful improvements, avoiding the pitfalls of guesswork.  

Myth #5: Experimentation is only for big companies 

There’s a myth that experimentation is only for the Amazons and Booking.com’s of the world—businesses with massive budgets and endless resources. But that’s not true. Even small businesses can benefit from experimenting, in fact, they arguably have more to gain. Smaller businesses with low traffic can experiment effectively by focusing on high-impact changes and using tools that make the process efficient.  

EasyChange proves that even with limited resources, experimentation is accessible. Their simpler approach helps them identify some useful improvements. However, ConsideredChange’s disciplined, data-driven strategy demonstrates that even small businesses can adopt rigorous experimentation practices to maximise their resources and achieve long-lasting results.  

Myth #6: Experimentation is about who’s right 

Not every experiment is going to give you a “win,” and that’s okay. Every test, whether it succeeds or “fails,” is an opportunity to learn. Maybe your test revealed that users prefer simpler designs or that a particular feature isn’t as valuable as you thought. These insights are just as valuable as a big win, guiding your next steps and making future efforts more informed.  

By weaving experimentation into the decision-making process with the precision and discipline of ConsideredChange, you can unlock the true potential of this powerful tool. The journey is about progress, not perfection—and the insights gained along the way will set the business on a path to sustainable success.

Myth #7: Any software will do

And remember, A/B testing software isn’t just for exciting new features or bold experiments—it’s equally important for bug fixes, compliance updates, and other mandatory changes. Testing ensures even the most routine updates don’t cause unintended harm. By embedding experimentation into your business, you’re setting the stage for steady growth, continuous learning, and better results all around. And with a tool that is trustworthy, easy to use, built for collaboration, and has all the features teams need, companies can begin to build or further optimise their experimentation culture. Dispel the myths, experiment smartly and unlock growth. Happy experimenting!

Curious about how A/B testing can work for you? Request a demo of ABsmartly today and see how we can help you make smarter, more confident decisions with real, measurable impact.