Someone tweaks a button label, the signup rate moves by a tenth of a percentage point, and the room splits into two camps: "it worked" and "that's noise." A/B testing exists to settle that argument before it costs another sprint of engineering time. The mechanics are simple — split traffic, measure, compare — but nearly all the difficulty lives in the decisions made before the first user ever sees the variant. Get those wrong and the statistics afterwards are just confident nonsense.

A hypothesis you could actually falsify

Most tests begin life as a hunch: "the new checkout flow should convert better." That is not testable in any useful sense. Push it into a shape with four parts: the change, the population, the metric, and the mechanism.

Because we saw drop-off at the shipping step, we believe moving cost estimates above the form will increase completed checkouts for first-time buyers, measured by checkout completion rate within seven days.

Two details in that sentence do most of the work. The metric is named before the test runs, and the mechanism explains why the change should move it. If the result later contradicts the mechanism, you learn something either way.

Pick one primary metric and commit to it in advance. Then add guardrails: page load time, refund rate, unsubscribe rate, support tickets. A variant that lifts clicks by making the page slower is not a win — it is a trade you may not want. Finally, write the decision rule while you are still ignorant. "We ship if the 95% interval for the primary metric excludes zero and no guardrail degrades." A rule written after you see the numbers is not a rule.

Sample size and power come first

Two error rates define every test. Alpha is the chance of declaring a winner when nothing changed. Power is the chance of detecting an effect that is genuinely there. The conventions of 5% and 80% are not laws of nature, but they are a reasonable default.

The arithmetic that surprises people

Four numbers determine your sample size: the baseline rate, the smallest effect you care about, alpha, and power. The second one does the damage. Detectable effect appears squared in the denominator, so halving the lift you want to detect roughly quadruples the traffic you need.

Take a 3% conversion baseline and a lift you would care about of 10% relative — 3.3% instead of 3%. A back-of-the-envelope calculation puts you near fifty thousand users per variant. If you only cared about a 20% relative lift, the requirement would fall to roughly a quarter of that. Many teams cannot reliably detect the effects they claim to care about, and pretending otherwise burns months.

Underpowered tests lie in both directions

When a low-power test does produce a significant result, that result tends to overstate the effect. Small studies clear the bar partly by luck, and luck inflates the estimate. This is why a "win" from a tiny pilot so often evaporates at full rollout. Underpowered tests do not simply miss real effects — they manufacture exaggerated ones.

What a p-value does and does not say

A p-value is the probability of seeing data at least as extreme as yours if the null hypothesis were true and the model were correct. That is all. It is not the probability the null is true, not the probability that the variant is better, and not the size of the effect.

• Report the effect size with a confidence interval, always. "Significant" without a magnitude is an incomplete sentence.
• Treat p = 0.06 and p = 0.04 as the same evidence. The threshold is a convention, not a cliff.
• Read the interval, not just its edges. A wide interval that excludes zero still contains effect sizes ranging from trivial to enormous.

The analysis sequence

1. Check for sample ratio mismatch. If the split is 48/52 when it should be 50/50, stop and find the bug.
2. Confirm the metric definition matches the code that logged it. Most surprising results are instrumentation.
3. Read the primary metric with its confidence interval, not the p-value alone.
4. Check the guardrails.
5. Look at pre-registered segments only. Anything you did not plan is a hypothesis, not a finding.
6. Apply the decision rule you wrote at the start.

Pitfalls that quietly ruin experiments

• Peeking. Checking daily and stopping at the first significant reading inflates false positives badly. Use a fixed horizon or a sequential method built for continuous monitoring.
• Multiple comparisons. Test twenty metrics and one will look significant by chance. Adjust, or pre-commit to a small set.
• Segment fishing. "It worked for mobile users in Germany" is often noise with a story attached.
• Novelty effects. Users click new things because they are new. Run long enough for behavior to settle, usually at least one full business cycle.
• Mix shifts. If the variant attracts different traffic, the overall average can move without any individual improving. Check the segment mix.
• Practical significance. A statistically clean 0.02% lift may not justify maintaining a second code path.

Turning a result into a decision

Almost no test ends in a clean yes or no. You get four outcomes: ship, don't ship, iterate, or run a better test. Document which one you chose, along with the hypothesis, the metric definition, the sample size, the duration, and the result. Six months later that note is the only thing standing between your team and re-running the same failed experiment.

When traffic is too thin to detect the effect you care about, say so plainly. A well-labelled inconclusive test is more honest than a two-week test that was never powered to answer the question. And when a small, real lift shows up in a high-traffic flow, resist the urge to dismiss it. Small and certain beats large and imaginary, especially when it compounds across millions of sessions.

Photo: pixelcreatures / Pixabay