Group Sequential Testing

Group Sequential Testing

Group Sequential Testing

Group Sequential Testing (GST) allows us to stop or ship a test early when we see a large effect, while also retaining the ability to detect smaller effects within a reasonable timeframe.

Group Sequential Testing (GST) allows us to stop or ship a test early when we see a large effect, while also retaining the ability to detect smaller effects within a reasonable timeframe.

Group Sequential Testing (GST) allows us to stop or ship a test early when we see a large effect, while also retaining the ability to detect smaller effects within a reasonable timeframe.

This approach combines elements from the modified Sequential Probability Ratio Test (mSPRT) and the classical fixed-horizon testing into a best-of-both-worlds method which is ideally suited for online controlled business experiments at scale.

This approach combines elements from the modified Sequential Probability Ratio Test (mSPRT) and the classical fixed-horizon testing into a best-of-both-worlds method which is ideally suited for online controlled business experiments at scale.

This approach combines elements from the modified Sequential Probability Ratio Test (mSPRT) and the classical fixed-horizon testing into a best-of-both-worlds method which is ideally suited for online controlled business experiments at scale.

Unlike mSPRT, Group Sequential Testing does not require us to drastically increase our sample size to be able to detect small effects. And unlike fixed-horizon testing, Group Sequential Testing does not require us to wait until the end of the experiment to make a decision. GST requires pre-determining the required sample size with a power calculation based on the minimum effect of interest, as well as specifying the number of analyses to be conducted throughout the duration of the experiment. Our intuitive user interface guides users through this process in an easy way.

Unlike mSPRT, Group Sequential Testing does not require us to drastically increase our sample size to be able to detect small effects. And unlike fixed-horizon testing, Group Sequential Testing does not require us to wait until the end of the experiment to make a decision. GST requires pre-determining the required sample size with a power calculation based on the minimum effect of interest, as well as specifying the number of analyses to be conducted throughout the duration of the experiment. Our intuitive user interface guides users through this process in an easy way.

Unlike mSPRT, Group Sequential Testing does not require us to drastically increase our sample size to be able to detect small effects. And unlike fixed-horizon testing, Group Sequential Testing does not require us to wait until the end of the experiment to make a decision. GST requires pre-determining the required sample size with a power calculation based on the minimum effect of interest, as well as specifying the number of analyses to be conducted throughout the duration of the experiment. Our intuitive user interface guides users through this process in an easy way.

In the case of a 30-day experiment where daily data review for decision-making flexibility is desired, the test can be configured with 30 analysis points. While this may slightly reduce statistical power compared to fixed-horizon testing, it greatly speeds up decision-making, leading to 20% to 80% faster decisions, as significance is commonly reached before the full sample is collected.

In the case of a 30-day experiment where daily data review for decision-making flexibility is desired, the test can be configured with 30 analysis points. While this may slightly reduce statistical power compared to fixed-horizon testing, it greatly speeds up decision-making, leading to 20% to 80% faster decisions, as significance is commonly reached before the full sample is collected.

In the case of a 30-day experiment where daily data review for decision-making flexibility is desired, the test can be configured with 30 analysis points. While this may slightly reduce statistical power compared to fixed-horizon testing, it greatly speeds up decision-making, leading to 20% to 80% faster decisions, as significance is commonly reached before the full sample is collected.

Importantly, Group Sequential Testing does not increase the false positive rate, and it can accelerate experiments by up to two times compared to the traditional mSPRT approach used in most tools on the market.

Importantly, Group Sequential Testing does not increase the false positive rate, and it can accelerate experiments by up to two times compared to the traditional mSPRT approach used in most tools on the market.

Importantly, Group Sequential Testing does not increase the false positive rate, and it can accelerate experiments by up to two times compared to the traditional mSPRT approach used in most tools on the market.

Try ABsmartly on A PoV

You can acquire a 60-day PoV (Proof of Value) to explore the features and experiment with the platform before committing to an annual subscription.

With ABsmartly, teams embrace experimentation

  • A/B Smartly helped clean up our AB testing process in many ways. It reduced the number of tools we used and gave us a simple SDK to consume within our applications.

    Kyle P.

    Senior Software Engineer

  • "It's a game changer for us"

    We are very serious experimenters and come from backgrounds where tens of billions of dollars of value were created through enterprise-grade UX experimentation programs.

    Scott Lake

    President Pharmacy

  • It scales for many experiments and goals/metrics, in isolated environments. The SDK is easy to work with (frontend and backend), and the Full-on mode saves time for winning experiments.

    David R.

    Software Engineer

  • "We love A/B Smartly"

    We moved from Google Optimize to A/B Smartly so that we could have full access and control of our data. We also want the ability to do server side/backend tests. We went with the onprem option and set up the service in our AWS account, the system has been running smoothly ever since.

    Mike Rizzo

    Principal Architect

  • A/B Smartly is easy to configure; It has a great dashboard that offers valuable insights at a glance. It helps me quickly set A/B experiments and get timely feedback on our hypothesis.

    Vince V.

    UX Designer

  • There's a lot to like about AB smartly; if I had to name it, it is optimized for high throughput: many experiments, many goals, multi-variant. The product covers everything I need to do currently without needing to solve anything myself or find ways of getting around it. It doesn't take a lot to get going and the team is extremely helpful.

    Peter C.

    Sr. Director of Engineering

  • Running A/B Smartly in our Google Cloud infrostructure side-by-side with our self-built data solutions gives us the best flexibility and we foresee AB Smartly as a platform that will scale with us for years to come.

    Sander Olislagers

    Head of Product Development & Marketing

  • The A/B Smartly platform is really easy to integrate within your current product/codebase and the support from the team is unsurpassed.

    Stefan B.

    CTO

With ABsmartly, teams embrace experimentation

  • A/B Smartly helped clean up our AB testing process in many ways. It reduced the number of tools we used and gave us a simple SDK to consume within our applications.

    Kyle P.

    Senior Software Engineer

  • "It's a game changer for us"

    We are very serious experimenters and come from backgrounds where tens of billions of dollars of value were created through enterprise-grade UX experimentation programs.

    Scott Lake

    President Pharmacy

  • It scales for many experiments and goals/metrics, in isolated environments. The SDK is easy to work with (frontend and backend), and the Full-on mode saves time for winning experiments.

    David R.

    Software Engineer

  • "We love A/B Smartly"

    We moved from Google Optimize to A/B Smartly so that we could have full access and control of our data. We also want the ability to do server side/backend tests. We went with the onprem option and set up the service in our AWS account, the system has been running smoothly ever since.

    Mike Rizzo

    Principal Architect

  • A/B Smartly is easy to configure; It has a great dashboard that offers valuable insights at a glance. It helps me quickly set A/B experiments and get timely feedback on our hypothesis.

    Vince V.

    UX Designer

  • There's a lot to like about AB smartly; if I had to name it, it is optimized for high throughput: many experiments, many goals, multi-variant. The product covers everything I need to do currently without needing to solve anything myself or find ways of getting around it. It doesn't take a lot to get going and the team is extremely helpful.

    Peter C.

    Sr. Director of Engineering

  • Running A/B Smartly in our Google Cloud infrostructure side-by-side with our self-built data solutions gives us the best flexibility and we foresee AB Smartly as a platform that will scale with us for years to come.

    Sander Olislagers

    Head of Product Development & Marketing

  • The A/B Smartly platform is really easy to integrate within your current product/codebase and the support from the team is unsurpassed.

    Stefan B.

    CTO

With ABsmartly, teams embrace experimentation

  • A/B Smartly helped clean up our AB testing process in many ways. It reduced the number of tools we used and gave us a simple SDK to consume within our applications.

    Kyle P.

    Senior Software Engineer

  • "It's a game changer for us"

    We are very serious experimenters and come from backgrounds where tens of billions of dollars of value were created through enterprise-grade UX experimentation programs.

    Scott Lake

    President Pharmacy

  • It scales for many experiments and goals/metrics, in isolated environments. The SDK is easy to work with (frontend and backend), and the Full-on mode saves time for winning experiments.

    David R.

    Software Engineer

  • "We love A/B Smartly"

    We moved from Google Optimize to A/B Smartly so that we could have full access and control of our data. We also want the ability to do server side/backend tests. We went with the onprem option and set up the service in our AWS account, the system has been running smoothly ever since.

    Mike Rizzo

    Principal Architect

  • A/B Smartly is easy to configure; It has a great dashboard that offers valuable insights at a glance. It helps me quickly set A/B experiments and get timely feedback on our hypothesis.

    Vince V.

    UX Designer

  • There's a lot to like about AB smartly; if I had to name it, it is optimized for high throughput: many experiments, many goals, multi-variant. The product covers everything I need to do currently without needing to solve anything myself or find ways of getting around it. It doesn't take a lot to get going and the team is extremely helpful.

    Peter C.

    Sr. Director of Engineering

  • Running A/B Smartly in our Google Cloud infrostructure side-by-side with our self-built data solutions gives us the best flexibility and we foresee AB Smartly as a platform that will scale with us for years to come.

    Sander Olislagers

    Head of Product Development & Marketing

  • The A/B Smartly platform is really easy to integrate within your current product/codebase and the support from the team is unsurpassed.

    Stefan B.

    CTO

Run and discuss experiments at scale.

Run and discuss experiments at scale.