Tools

False Positive Risk Calculator for AB Tests

A p-value of 0.05 does not mean a 5% chance the result is a false positive. Set your parameters and see the true probability your decision is wrong, in both directions.

Test result

Observed p-value

Or drag the chart

Significance level α

0.050

Statistical power (1−β)

80%

Test design

→ MDE Calculator

Power calculation

Hypothesis type

Bayesian context

Prior P(H₁) · chance idea is real

50%

Not the historical win rate (that's conditioned on significance)

Risk Curve

// drag the dot or use the p-value slider above

False Positive Rate

5.0%

by design (α = 0.05)

your call

False Positive Risk

p-equalsip-equalsWhen to useYou have a specific observed p-value and want to know the risk for this exact result.What it assumesThe true effect, if real, is exactly at your MDE (point alternative at μ₁). Uses the likelihood ratio at the exact observed z-score.When NOT to useAvoid when you have no reliable MDE estimate. The result is sensitive to the assumed effect size.29%

Likelihood ratio at the exact p-value. Point alternative at μ₁ (your MDE). Colquhoun (2019)

p-less-thanip-less-thanWhen to useYou want the long-run false positive rate across your entire testing program, not just this test.What it assumesAll significant tests are treated equally regardless of their exact p-value. Effect exists at the power level you specified.When NOT to useAvoid for interpreting a single result. It ignores the exact p-value and describes the program average, not this specific test.5.9%

Program-level rate: α·π₀ vs power·π₁. Does not depend on exact p-value. Kohavi, Deng & Vermeer (2022)

SBB minimumiSBB minimumWhen to useYou want a lower bound on FPR that does not depend on any assumed effect size or sample size.What it assumesNothing about the alternative. Maximises the Bayes factor over all simple alternatives. Valid for any p < 1/e ≈ 0.37.When NOT to useAvoid treating it as a point estimate. It is the best-case (minimum possible) FPR, not the expected FPR.29%

Best-case lower bound over all alternatives. Depends only on p-value, not n. Sellke, Bayarri & Berger (2001)

chance your "winner" is a fluke

False Negative Risk

Posterior probability of a missed true effect at this exact p-value. Colquhoun (2019)

Program-level rate: (1−power)·π₁ vs non-significance. Does not depend on exact p-value. Kohavi, Deng & Vermeer (2022)

chance you're killing a real winner

Diagnosis & Recommendation

Proceed with caution: 29% false-positive risk at p = 0.0497. Significant, but not conclusive. Consider replicating or raising the bar before a permanent rollout.

Replication: Combine Multiple Tests

Don't decide on one borderline test. Replicate and combine. Sidedness (two-sided) and power (80%) are inherited from the parameters above.

0.0497

current test (from above)

Add a new p-value to meta-analyze.

Sources: Kohavi, Deng & Vermeer (2022). A/B Testing Intuition Busters. KDD '22. · Colquhoun (2019). The False Positive Risk. The American Statistician. · Sellke, Bayarri & Berger (2001). Calibration of p-Values. The American Statistician. · Georgiev (2023). False Positive Risk in A/B Testing. analytics-toolkit.com