Learn
Updated 2026-03-07 UTC
A/B Test Significance — Interpreting p-values, Lift, and Sample Size — GetCalcMaster
A/B test workflow for conversion rates: compute lift, run a two-proportion test, interpret significance vs practical impact, and plan sample size.
An A/B test is a two-group experiment. “Significance” answers whether the observed difference is likely to be noise; it doesn’t automatically mean the change is worth shipping. This guide focuses on interpretation and planning.
Important: Educational. For production experimentation programs, consider sequential testing, CUPED/variance reduction, and pre-registration of analysis plans.
What this calculator is
The Statistics Calculator is an interactive tool inside GetCalcMaster. It’s designed to help you explore scenarios, understand formulas, and document assumptions.
Key features
- Separates statistical significance from business impact
- Shows how lift relates to absolute vs relative differences
- Highlights multiple metrics and multiple comparisons pitfalls
- Connects to power and sample size planning
Formula
Absolute lift: Δ = pB - pA
Relative lift: (pB - pA) / pA
Significance test: two-proportion z test (see linked guide)Quick examples
A result can be significant but tiny (large n). Decide based on expected value, not only p-value.A large lift can be non-significant if the test is underpowered (small n).If you look at results repeatedly (“peeking”), your false positive rate increases unless you use sequential methods.
How to use it (quick steps)
- Compute conversion rates pA and pB and the absolute difference Δ = pB − pA.
- Compute relative lift = (pB − pA)/pA (only if pA>0).
- Run a two‑proportion test (or compute a CI for Δ).
- Check practical impact: does Δ matter at your traffic and revenue scale?
- If testing many metrics or variants, apply a multiple-comparisons correction.
Related tools and guides
Tools
Featured guides
Deep, human-written guides focused on accuracy, verification, and reproducible workflows.
FAQ
Is this calculator official?
No. GetCalcMaster provides educational estimates and learning tools. Always verify against official definitions, documents, or professional advice.
Do you store my inputs on the server?
No. Calculations run locally in your browser. Optional remember/restore features (if enabled) use local browser storage.
Tip: For reproducible work, save your inputs and reasoning in Notebook.