I Helped Build the Experimentation Engine at Booking.com: Here's What I Learned

When people ask me what made Booking.com grow the way it did,consistently outpacing the S&P 500 for more than a decade, most expect a silver bullet: a brilliant product idea, a genius hire, or a lucky market bet. They’re surprised when I tell them: it was A/B testing. But not in the “run 100 tests a day” kind of way you’ve read about. That’s the result, not the reason. Let me explain.

Booking.com Didn't Just Do Experimentation. We Became an Experiment-Driven Company.

When I joined Booking.com, we didn’t yet have a mature experimentation platform. I had the opportunity to help build it: the infrastructure, the tooling, the processes, and most importantly, the culture. It wasn’t glamorous. Most days, it was debugging messy data pipelines or building guardrails so that product managers could test safely without breaking the site.

What we were doing wasn't just tech, it was enabling a fundamentally different way of thinking.

We believed decisions shouldn’t be made by the highest-paid person in the room. They should be made by data. And that meant creating a system where every meaningful product decision got tested, no matter how small or “obvious” it seemed.

You’d be surprised how often the “obvious” thing turned out to be wrong.

Here’s What People Miss About Experimentation

When folks outside Booking.com hear that we were running tens of thousands of experiments per year, they assume that velocity alone is what mattered. That if you just scale the number of tests, you'll get the same outcomes.

That’s a misunderstanding.

The secret wasn’t the number of tests. It was that experimentation became our default mode of decision-making.

Every engineer, product manager, designer, marketer, copywriter, they were trained to ask: “How do we know if this will work? Should we test it first?”

This mindset was embedded into our development workflow. Features weren’t “done” without being tested tested. And we didn’t just test what we hoped would work, we also tested what we feared wouldn’t, as long as it went in the direction we wanted the website to go. That’s where the real insights came from.

It was a form of organizational humility. We admitted we didn’t know what would work until we saw real user behavior.

Scaling Experimentation Is Hard, But That’s Why It’s Worth It

If you’ve ever tried to build an experimentation program, you know it’s not just about tools. Tools are easy, culture is hard.

Experimentation threatens egos. It forces leadership to be okay with being wrong - a lot. It exposes how often we bet big on ideas that don’t move the needle. It requires engineering and design teams to slow down just enough to measure impact before they roll out the next shiny thing.

And maybe hardest of all: most tests don’t “win.”

At Booking.com, a majority of experiments didn’t produce statistically significant uplifts. But that didn’t mean they failed. Every test, positive, negative, or insignificant, gave us more signal, more clarity, and fewer assumptions clouding our judgment.

Why We Started ABsmartly

After Booking.com, I saw companies trying to copy the testing volume without copying the mindset. They wanted the growth curve but weren’t ready for the cultural shift.

So We built ABsmartly, not just as an experimentation platform, but as a system for helping organizations internalize the same principles we lived by at Booking.com:

• Those decisions should be evidence-based, not opinion-based.

• That infrastructure should enable anyone, not just data scientists, to test safely.

• That velocity without discipline is chaos.

• That data literacy and organizational buy-in are more important than any dashboard.

• That clear communication is crucial

We built ABsmartly so teams don’t just run more tests; they run better tests. More trustworthy, faster to deploy, easier to analyze, and safer at scale.

What Would It Take to Make Experimentation Your Default?

You may not have Booking.com-level traffic, but you don’t need it. What you need is commitment.

Commitment to:

• Equip your teams with tooling that doesn’t get in their way.

• Accept that many of your best ideas will flop, and that’s okay.

• Build processes that make testing part of “done,” not an afterthought.

• Treat experimentation not as a tactic or a role, but as a core business function that everybody in the company...
  …relies on to validate assumptions and reduce risk.
  …integrates into everyday workflows and thinking.
  …embraces to continuously learn and improve.
  …champions to stay agile and adaptive
  …treats as part of the company’s DNA

Companies that do this become more agile, more customer-focused, and more resilient to market shifts. They ship smarter, fail faster, and learn more in a quarter than most teams do in a year.

Ask Yourself This

How many experiments did your team launch last year? How many of those actually changed how you work or think?

Now ask yourself: What would it take to double that this year?

Not just in quantity, but in quality, in insight, in organizational impact.

If you're ready to turn experimentation from a project into a pillar of your company’s growth strategy, that’s what ABsmartly is here to help with.

Jonas Alves
Founder, ABsmartlyFormer Experimentation Lead, Booking.com