Understanding Significance

Welcome to the second module. Now that you know how a real A/B test is structured, it's time to tackle the key question: when can we actually trust a result? In this module, you'll learn to read significance, p-values, confidence intervals, and sample size in a way that prevents false rollouts. The goal isn't to become a statistician. It's to avoid expensive mistakes.

By the end of this module, you'll evaluate test results the way an experienced experimenter does. You won't just check the uplift; you'll also assess the confidence level and the business relevance. These three skills protect you from the most common mistake in A/B testing: an apparent winner that's actually just noise.

Two quick self-checks before we get into the details. First, take a guess: what does a high significance level actually tell you? Then think one step further: a test can show a large uplift and still have no business case. Why? These two questions lead us straight to the core concepts of this module.

A lot of people are intimidated by the p-value because it sounds abstract. But it's really just a number that describes how likely the observed result would be if nothing had actually changed. A p-value of 0.05 means: if the variant has no real effect, we'd still see a difference this large in 5 out of 100 cases just by chance. Below that threshold, we treat the effect as real.

At JDKRUEGER&CO, the threshold is fixed: 95%. That's not dogma. It's a risk management standard. Any lower threshold would produce too many false rollouts. Imagine rolling out at 80% significance. That means one in every five apparent winners is actually just chance. For real budgets, that's too much risk.

The point estimate alone can be misleading. An interval of -1% to +7% tells you the effect could even be slightly negative. That's not a solid basis for a rollout. That's why we always read the full interval, not just the midpoint. If zero is inside the interval, we can't claim there's a real effect.

This is the classic dilemma. The uplift looks tempting, but the data isn't there yet. Stopping early would mean celebrating a result that has an 8% chance of being pure noise. Cases like this are exactly what separate professional testers from teams that roll out based on optimism. Patience here is the economically rational choice.

This is one of the most expensive mix-ups in A/B testing. Significance is not a quality stamp for revenue. You need to check both dimensions: certainty and effect size. A test can be statistically sound and still have no business case. That's why a cost-benefit analysis always belongs alongside the p-value.

Before you roll out a winner, ask yourself: Is it worth it? A test can be statistically sound and still have no business case. The MDE helps you settle that question upfront. It's the smallest change that still matters for your shop. Anything below that is measurable, but not worth rolling out.

These three misconceptions cost businesses money every year. The p-value refers to the data under the assumption that nothing happened. It says nothing about the probability of your hypothesis. A non-significant result doesn't mean the variant is neutral. It just means there isn't enough data. And a small p-value says nothing about the size of the effect.

Significance is a statement about this sample and this time period. External factors like seasonality, competitive responses, or new campaigns can change the effect later on. A 5% error probability remains. The effect is an estimate with a confidence interval, not an absolute truth.

Imagine asking five people about the political mood. Then you ask five hundred. The second result is more reliable. The same applies to conversions in an A/B test. The more data you collect, the narrower the interval gets and the more reliable the estimate becomes. That's why tests need time and traffic.

Quick check-in. The p-value tells you something about the data, not about the hypothesis. The interval shows where the true effect most likely falls. And significance alone is no reason to roll out. It has to be accompanied by practical relevance. Keep these three points in mind and you'll avoid the most expensive mistakes.

We see this scenario regularly. A large uplift at still-insufficient significance tempts you into action. The disciplined approach is to let it run longer, or accept that the result stays inconclusive. 88% sounds high, but it still means a 12% chance the effect is random.

Here you can see the connection between statistics and the business case. The MDE isn't just a calculation variable. It's the smallest improvement that still makes financial sense for you. If your test delivers only two percent uplift but the MDE requires fifteen percent, the result is neither statistically nor economically convincing.

These terms come up in every sample size calculator. You don't need to memorize them, but you do need to understand them. Alpha protects you from false wins. Power protects you from missing real opportunities. If your test has a power of eighty percent, you'll detect a real effect in four out of five cases. In one out of five, you'll miss it.

Peeking is the silent killer in A/B testing. Every additional look at the data is like running a new test round. Eventually, you'll inevitably find a false winner. That's why we set the runtime and stopping rules before the test begins. If you're checking daily, you're not running an experiment anymore. You're gambling.

Take a minute. The p-value is below 0.05, so the statistical threshold is cleared. The missing pieces are the confidence interval, the MDE, and the implementation costs. Without those three data points, you don't know whether the eight percent uplift is economically meaningful. That gap is exactly what leads to bad rollout decisions.

These four criteria form our standard decision tree. If all of them are met, you can roll out with high confidence. If one is missing, proceed with caution. This rule protects you from the two most common mistakes: rolling out too early and waiting too long. Both cost money.

Remember this: significance is a gatekeeper, not a quality seal. It tells you an effect is probably real. Whether it's economically valuable is a separate question, one you answer with MDE, costs, and risk. Anyone who checks both makes better decisions than ninety percent of the market.

These three habits make the difference: uplift plus p-value plus confidence interval, MDE as an economic threshold, and disciplined stopping rules. Apply them in your next test and you'll already be operating more professionally than most shops. In the next module, we'll look at how to calculate the right test duration and plan your traffic.

At JDKRUEGER&CO, no test is declared a success just because it feels good. We document p-values, confidence intervals, MDE, and rollout risk. In the next module, we'll look at how to calculate the right test duration and plan your traffic.