Successfully Navigating Experimentation with ABsmartly's Health Check Panel, March Features Release Blog

Being agile is a key part of building a business, and the difference between huge success and mediocre results often hinges on the ability to quickly understand and act on data. This is where ABsmartly's Health Check Panel comes into play, improving the way businesses manage their experimentation efforts. With its "single pane of glass" dashboard, the Health Check Panel now available inside the ABsmarrtly experimentation platform provides a comprehensive, real-time view of all alerts and issues, helping users to proactively manage their online experiments, increase their visibility, and save precious time and resources.

A Bird’s Eye View of the Quality of your Online Experiments

ABsmartly's Health Check Panel is not just part of what we do — it's designed to be your experimentation copilot. Offering a holistic view that goes beyond merely showing you the data, it acts as an important function in the experimentation platform in ensuring that your experiments are both high quality and efficient. Our Panel is designed to give users a bird's eye view of their experiment’s quality, highlighting the health and performance of each test in real time. The visibility isn’t just about pinpointing issues; it's about allowing you to take swift decisive action.

Real-Time Alerts and Checks: Your Early Warning System

The Health Check Panel's strength lies in its suite of alerts and checks; each is tailored to identify potential issues before they escalate. Here's a look at some of the key features:

• Sample Ratio Mismatch Check: Imagine setting up an experiment to split traffic 50/50, only to find a 70/30 split due to a setup or execution error. The Health Check Panel detects such discrepancies in real time, using a chi-squared test to alert you to the mismatch. This gives you the chance to fix issues promptly, safeguarding the integrity of your experiments.

  
  

• Audience Mismatch: Tailoring experiments to specific audience segments is an important part of the experimentation process. Our Health Check Panel ensures that the right people see the right experiments and lets you know if there are any deviations from your targeted audience setup.

  
  

• Variable Conflicts: Our variables allow you to run experiments with a configuration payload and then set up experiments without having to touch the code. The Health Check Panel identifies any conflicts in these variables, ensuring seamless experimentation across different audience segments without the need for developer intervention. So you don’t have random variables conflicting with each other for the same audience in different experiments. 

Other Platform Features Newly Released

Full expression power for segment filters and actionable insights

When a particular audience has a different behaviour from the average, you might want to drill down into the data to understand why. Why are there different behaviours for different audiences? With ABsmartly, you can take action on these insights immediately and debug directly in the platform, there is no need to export the data or use Google Analytics or other tools to do the analysis. Ultimately, this feature also lets you use the insights to better plan the next steps or follow-up experiments. 

Let’s look at an example. In one experiment meant to improve site speed, we noticed that the average load time of the site actually got worse. This was the opposite of what we were expecting. After drilling down into the segments by country we noticed that the variant had a lot more users in the countries where the website was slower. What actually happened was that the website got so much faster that users that before could not even load the page now can browse the site as well, but because they have slower page load times the total average in that variant looks worse than the average page load times in the control. This is because the control variant is missing all those users that didn't wait for the page to load.

Option to use last-seen instead of first-seen

Allow users to optionally segment the data based on the most recent attributes of the customer instead of the first ones. Based on customer feedback, we listened and wanted to improve this feature and make it more flexible.

Sample Size Reached

Figuring out when an experiment has gathered sufficient data can be a guessing game. This alert lets you know when the sample size is enough for you to take action. You don’t have to continuously check to see if the experimentation is finished. You can focus on other tasks and trust the panel to let you know when the experiment has reached maturity. This greatly streamlines not only your decision-making but also your reporting processes.

Boundary Crossed

Real-time alerts for significant deviations help prevent false positives while making sure your experiments really are providing valuable insights. For example, if a test crosses its efficacy boundary it means it has gathered enough information to allow you to stop it and implement the winning variant with confidence.

Code Cleanup Needed

Post-experiment code cleanup is often overlooked, so this alert reminds you to tidy up your codebase by removing the code paths that are not used anymore.

The Strategic Advantage of Proactive Management

By integrating these real-time alerts and checks, the Health Check Panel does more than just monitor; it offers a strategic advantage. It allows teams to act quickly, addressing issues before they impact the experiment's outcomes. This proactive approach saves time and resources and boosts confidence in the quality and reliability of your experimentation efforts.

ABsmartly is designed to streamline the experimentation process, making it more efficient and effective by allowing teams to run thousands of simultaneous A/B tests across their entire infrastructure​​. This comprehensive approach ensures that businesses can use experimentation to drive growth without having to make huge financial investments. Our vision is to allow wider access to experimentation tools. We allow you to conduct A/B testing more rapidly and effectively, using one single tool and allowing teams to respond more quickly to emerging data and insights, ultimately leading to faster iteration and optimisation cycles.

The strategic advantage of this proactive management approach lies in its ability to democratise experimentation across an organisation. By enabling designers, engineers, and product teams to control A/B testing directly, ABsmartly facilitates a culture where experimentation isn’t some sort of dark art that a center of excellence knows about, but a fundamental aspect of the entire product development lifecycle​​. By providing an overview of key metrics and experiment statuses, we want to help you create a culture where data-driven decision-making is the norm  — not the exception.

A Clear Path Through the Experimentation Woods

Have you ever heard the saying about not being able to see the woods for the trees? Our aim with the Health Check Panel is to make it easy for companies to access the information they need to iterate and move forward. We don’t want to give you a whole forest of meaningless data; we want to give you one single tool that will give you clarity and control. Think of the Health Check Panel as that one huge oak tree in the middle of the forest that helps you navigate the experimentation process. It transforms the daunting task of managing numerous experiments into a streamlined process. 

The Health Check Panel makes experimentation not just manageable but a strategic asset by offering a solution that not only anticipates the needs of its users but helps them through the experimentation process. Our Health Check Panel is your trusty copilot, making it easy to proactively manage experiments; make smart, informed decisions; save precious time; and democratise the process of A/B testing.