Statistical Significance
Statistical Significance is a measure of how confident you can be that an observed result reflects a real effect rather than random chance.
Also known as: statistical confidence, stat sig, test significance
Statistical Significance tells you whether the difference you see in a test, such as one landing page converting better than another, is likely a real effect or could plausibly be explained by random variation. It is the formal language for separating signal from noise in any experiment or comparison.
What Statistical Significance Means
Statistical Significance is the output of hypothesis testing, which produces a p-value representing the probability of seeing your result if there were truly no effect. A result is typically called significant when that probability falls below a threshold, commonly 5 percent, giving you reasonable confidence the effect is real. The 5 percent threshold is convention, not law, and other thresholds may make sense depending on the cost of being wrong. Some teams use stricter thresholds (99 percent) when the cost of being wrong is high, or looser thresholds (90 percent) when speed matters more than certainty.
How Statistical Significance Works
A p-value is the probability of observing a result at least as extreme as yours if there were no real effect. A low p-value suggests chance is an unlikely explanation for the result, but it does not tell you the size or importance of the effect. It is a measure of evidence against the null hypothesis. Power is the probability that a test will detect a real effect if one exists; well-designed tests have at least 80 percent power. A two-tailed test checks whether the variant is different from control in either direction; a one-tailed test checks only one direction and needs smaller samples but only works when the direction of interest is committed in advance.
Common Pitfalls and Misconceptions
The widespread misconception is that significance equals importance. A test can be statistically significant yet show a tiny, commercially meaningless lift, or it can miss a real effect because the sample was too small. Significance is about confidence in the direction; effect size is about whether the magnitude matters. Both must be read together, and reporting only one is the single most common error in marketing analytics. The second pitfall is peeking and stopping early when a result looks significant: this inflates false-positive rates from 5 percent to 25 percent or higher, which is why disciplined experimenters never stop a test based on partial data.
Statistical Significance in Practice
The practitioner discipline is pre-specifying the test design before running it. Decide in advance the minimum effect size worth detecting, the sample size required to detect it at 95 percent confidence, and the test duration that achieves that sample. Then run the test until the duration completes, regardless of how the interim numbers look. The protocol matters more than the math. For B2B teams with low conversion volume, the right approach is to pre-specify the minimum effect size worth detecting and accept upfront that the test will not reach Statistical Significance, treating results as directional. Underpowered tests dressed as significant are the most common B2B testing error.
Frequently asked questions
-
What does a p-value actually mean?
A p-value is the probability of observing a result at least as extreme as yours if there were no real effect. A low p-value suggests chance is an unlikely explanation for the result, but it does not tell you the size or importance of the effect. It is a measure of evidence against the null hypothesis, not a measure of how big or meaningful the effect is.
-
Why is 95 percent confidence the common standard?
It is a widely adopted convention, not a law. A 95 percent confidence level means a 5 percent chance of a false positive. Some teams use stricter thresholds (99 percent) when the cost of being wrong is high, or looser thresholds (90 percent) when speed matters more than certainty. The threshold should reflect the decision's stakes.
-
Can a result be significant but still not matter?
Yes. With enough data, even a trivial difference can become statistically significant. A 0.1 percent conversion rate lift can be highly significant on a million-visitor test and commercially worthless. That is why you should always check the effect size to confirm the result is large enough to act on, not just the p-value.
-
What is statistical power?
Power is the probability that a test will detect a real effect if one exists. Underpowered tests, usually from small samples, frequently miss genuine effects and produce inconclusive results. A well-designed test has at least 80 percent power to detect the minimum effect size that would justify acting on the result.
-
Why should I avoid peeking at results early?
Repeatedly checking a test and stopping as soon as it looks significant inflates false positives substantially, often from 5 percent to 25 percent or higher. You should define the sample size and duration in advance and wait until the test completes before drawing conclusions. The protocol matters as much as the math.
-
What is the difference between one-tailed and two-tailed tests?
A two-tailed test checks whether the variant is different from control in either direction. A one-tailed test checks only whether it is better (or worse) in a specific direction. One-tailed tests need smaller samples but only work when the direction of interest is committed to in advance. Switching to one-tailed after seeing the data is a common form of p-hacking.
-
Should B2B marketers worry about statistical significance?
Yes, but pragmatically. Many B2B tests have such low volume that achieving significance requires impractical durations. The right approach is to pre-specify the minimum effect size worth detecting and the test duration required, then either run the test to completion or acknowledge upfront that the test will not reach significance and treat results as directional. Underpowered tests dressed as significant are the most common B2B testing error.