Is your A/B test result real?

Your test results

Conversions are visitors who did the thing you are measuring (bought, signed up, clicked) counted once each. The planned share is what your tool was set to send to control, used only for the sample ratio mismatch check. Enter how many times you looked at the results and considered stopping; 1 means you looked once, at the end.

Enter your numbers, then click Analyze.

Result

Testing two prices? Compare revenue per visitor, not conversion

A higher price almost always converts worse. What matters is whether it earns more per visitor. Enter each price and how many visitors saw it and bought.

Uses the significance level chosen above. Two-sided test.

More than one variant? Test A/B/n without fooling yourself

One line per arm: name, visitors, conversions. The first line is the control. Each variant is compared with control, and p-values are Holm-adjusted so testing several variants does not make a false winner likely.

Uses the significance level chosen above. Two-sided test.

Plan a test: how many visitors do you need?

Uses the significance level chosen above. Two-sided test.

Or: what lift can your traffic detect?

How it is calculated

Fixed-sample test: pooled two-proportion z-test, two-sided. The interval on the difference uses the unpooled standard error; the interval on relative lift uses the log risk ratio (Katz). Sample ratio mismatch: chi-square goodness of fit on the two group sizes against your planned split; a p-value below 0.001 is flagged as a likely broken test. Bayesian: uniform Beta(1,1) priors, 20,000 seeded posterior draws; reports the probability the variant is better, the expected loss from choosing each side, and a 95% credible interval on relative lift. Always-valid: a mixture sequential probability ratio test (mSPRT) with a normal mixing distribution whose spread is 10% of the control rate; its p-value and interval remain valid at every look, so they are wider than the fixed-sample ones. Peeking: the false-positive rate for stopping the first time p < your level across evenly spaced looks, from 20,000 seeded simulations. Sample size: the standard two-proportion formula with unpooled variance under the alternative. Price test: revenue per visitor is conversion rate times price; the difference is tested with a delta-method z-test (variance price²·p(1−p)/n per group), the Bayesian chance that B earns more uses the same seeded Beta draws scaled by each price, and the break-even conversion is the control rate times the price ratio.

Not modelled: more than two variants (correct for multiple comparisons yourself), revenue per visitor with varying order values (the price panel assumes one fixed price per group), retention after purchase, variance reduction, novelty effects, and weekly seasonality. Unverified: checked against published reference values and simulation in unit tests only, not against a commercial platform.

Common questions

How do I know if my A/B test result is statistically significant?

Compare the p-value from a two-proportion z-test with your significance level, usually 0.05. A p-value below 0.05 means a difference this large would show up less than 5% of the time if the two versions truly converted the same. This tool also shows the confidence interval for the difference, which tells you how big the effect could plausibly be, not just whether it is zero.

How many visitors do I need for an A/B test?

It depends on your baseline conversion rate, the smallest lift you care about, your significance level and your power. As a guide, detecting a 10% relative lift on a 5% baseline at 95% confidence and 80% power takes about 31,000 visitors per group. Use the planner above to enter your own numbers and see how many days that takes at your traffic.

What is sample ratio mismatch (SRM)?

If you planned a 50/50 split but the groups came out 5,000 and 4,500, something is wrong with assignment or tracking. A chi-square test on the group sizes flags splits that are too lopsided to be chance. When SRM fires, the conversion comparison cannot be trusted, so fix the cause first.

Why is peeking at an A/B test a problem?

A fixed-sample p-value is only valid if you look once at a sample size chosen in advance. If you check repeatedly and stop the first time p drops below 0.05, the real false-positive rate climbs to about 14% with five looks and 19% with ten. The always-valid p-value and interval this tool shows stay valid however often you look, at the cost of being more conservative.

What does the Bayesian result mean?

It uses a uniform Beta prior on each conversion rate and reports the probability that the variant's true rate is higher than the control's, plus the expected loss if you choose each option. A 96% probability means that, given the data and the prior, the variant is better with about 96% credibility. It is not the same number as one minus the p-value.

Which result should I trust, frequentist, Bayesian or always-valid?

On a test you planned for a fixed sample size and looked at once, the ordinary p-value and interval are right. If you monitored the test as it ran, use the always-valid result. The Bayesian probability is a useful second view of the same data, but decide your stopping rule before you start whichever you use.

How do I know if a price increase worked in an A/B test?

Compare revenue per visitor, which is conversion rate times price, not conversion alone. A higher price usually converts worse, so a conversion test will often call it a loss even when it earns more. The price panel above tests the difference in revenue per visitor and shows the break-even conversion: raising $29 to $39 can lose up to about 26% of buyers before revenue per visitor falls. It looks only at first-purchase revenue, so also check refunds and churn before rolling out.

Is my conversion data uploaded anywhere?

No. You type four numbers and everything is computed in this browser tab. There are no accounts, cookies or analytics vendors.

Testing ads across regions or stores instead of page visitors? The ad incrementality test analyzer reads a geo holdout and measures the lift with an interval.