Bottom line: Statistical significance answers, “Given the specified model and the null hypothesis of zero difference, how incompatible are these data?”; effect size answers, “How large is the observed difference?”; confidence intervals answer, “Under the same model, what range of differences remains compatible with the data?” None of these alone tells you whether to launch. Decisions also require defining a minimal meaningful difference before looking at the results, alongside cost, risk, sample representativeness, measurement error, and causal design. Large samples can make trivial differences highly significant, while small samples can leave important differences uncertain.
The p-value is not the probability that the conclusion is true
Assume the null hypothesis states that versions A and B have identical completion rates in the target population. The p-value is the probability, under that null plus the sampling model and other assumptions, of obtaining the observed data or something more extreme. It is not the probability that the null is true, nor the probability that the result is due to chance, nor the probability of replication in a future study. Evidence from p = 0.049 and p = 0.051 is nearly identical; crossing the 0.05 threshold does not turn the former into “effective” and the latter into “ineffective.”
The p-value depends on the full model: independence, sampling or assignment method, variable coding, variance estimation, and pre-specified analysis all matter. When questionnaires come from quota samples, cluster delivery, or weighted samples, computing as if they were simple random samples yields incorrect standard errors. Question wording, non-response bias, or data-cleaning choices are systematic errors; increasing sample size reduces sampling fluctuation but does not automatically fix these biases.
Example 1: 200,000 people make a 1 percentage point difference highly significant
In a login-flow experiment, two groups—A and B—each receive 100,000 randomly assigned users. A has a 75% completion rate, B has 76%, an absolute difference of 1 percentage point, and a relative improvement of (76% − 75%) ÷ 75% ≈ 1.33%. Using the standard large-sample approximation for two independent proportions, the standard error of the difference is √[75%×25%÷100000 + 76%×24%÷100000] ≈ 0.192 percentage points. The 95% interval is 1 ± 1.96×0.192, or about 0.62 to 1.38 percentage points; z ≈ 5.20, two-sided p < 0.000001.
These data are extremely incompatible with “zero difference” and the direction is clear, but this does not mean that 1 percentage point is sufficiently important. If the team set a priori that only a gain of at least 3 percentage points justifies the migration cost and potential risk, then the entire interval falls below 3 points: the result is statistically precise, yet misses the preset business threshold. If there are tens of millions of logins daily, 1 percentage point might still yield huge absolute value; importance arises from business scale and consequences, not from a p-value label.
Example 2: With 500 people, an 8 percentage point difference is still directionally imprecise
In another concept test, groups A and B each have 250 respondents; purchase consideration is 50% and 58%, respectively—an absolute difference of 8 percentage points, or a 16% relative improvement. The standard error of the difference is approximately √[50%×50%÷250 + 58%×42%÷250] = 4.44 percentage points. The 95% interval is roughly 8 ± 1.96×4.44, i.e., −0.7 to 16.7 percentage points; z ≈ 1.80, two-sided p ≈ 0.07.
Calling this outcome “no effect” is wrong: the current data remain compatible with a slight adverse effect, with a negligible difference below 3 percentage points, and with a large positive effect. The appropriate decision might be to continue with a pre-planned extended sample, to check key subgroups and measurement quality, rather than declaring a winner. Contrasting with Example 1, a larger point estimate does not mean more precise evidence, nor does higher significance mean a more important effect.
How to read confidence intervals—rather than checking only whether they cross zero
The strict frequentist definition of a 95% confidence interval is: if one repeatedly samples by the same design and constructs intervals the same way each time, about 95% of those intervals will cover the true parameter. For the single interval already computed, one should not simplify it to “there is a 95% probability that the true value lies inside.” A more practical reading is to place the interval on the same axis as the zero difference and the business threshold, and see which consequences remain plausible given the data.
Set the minimal meaningful difference at ±3 percentage points. Four decision states emerge. If the interval lies entirely between −3 and +3, the result is sufficiently precise to be near the negligible zone. If the interval crosses zero and also crosses +3 or −3, both direction and importance are unclear. If the interval excludes zero but still crosses ±3, the direction is fairly clear but whether it is worth acting on remains uncertain. Only when the entire interval lies above +3 can you support “positive and reaching the preset threshold.” This retains much more decision-relevant information than the binary “significant / not significant” labeling.
Report interpretable effect sizes on the original scale first
For proportions, report percentage-point differences first and relative ratios as a complement; for means, report the raw-scale score difference; for times, report the median difference or a distribution-appropriate ratio. Standardized mean differences, correlations, or odds ratios are useful for cross-scale comparison and model analysis, but readers may not directly map them onto business consequences. The “small, medium, large” thresholds are not transferable across fields mechanically; the same 0.4 standard deviation can mean something entirely different in safety incidents, brand attitudes, or internal processes.
Effect sizes also need intervals. B being 8 percentage points higher than A is only a point estimate in the sample; −0.7 to 16.7 shows the current precision. Relative improvements depend heavily on the baseline: going from 1% to 2% is a 100% relative improvement but only a 1 percentage point absolute difference; from 50% to 55% is only a 10% relative improvement but a 5 percentage point absolute difference. Reporting both absolute and relative scales reduces overstatement.
Non-significance does not prove equivalence
If the research goal is to show that two versions are similar enough to be considered equivalent, you must pre-define an acceptable margin and run an equivalence test; if the goal is to show that the new option is not worse than the old by more than a specified amount, use a non-inferiority design. Failing to reject the null in an ordinary test may simply reflect insufficient sample size or measurement noise. In Example 2, the interval is as wide as 17.4 percentage points—clearly incapable of supporting “the two options are the same.”
Likewise, a significant result does not prove causation. A randomized, well-executed experiment can bring the difference closer to an average treatment effect; in a cross-sectional survey, “high-frequency users are more satisfied” could stem from selection, reverse causation, or common causes. Tests quantify only the random uncertainty of a particular estimator; they do not upgrade correlation to causality.
Multiple comparisons create selectable asterisks
Even in a simplified setting with no true differences and independent tests, running 20 tests each at a 5% threshold yields a familywise type I error rate of 1 − 0.95²⁰ ≈ 64.2%, far above 5%. Real-world metrics are correlated and often shift, but post-hoc browsing of dozens of slices and reporting only the smallest p-value still exaggerates evidence. Pre-specify primary metrics, comparisons, and directions; label explorations explicitly; and control familywise error or false discovery rate according to the problem. Validate again on fresh samples.
Selective cleaning, repeatedly trying different groupings, or switching to one-sided tests after seeing the data also add analytic degrees of freedom. Retain the analysis plan, code versions, and all major results—even when not significant. Transparent reporting is not an optional virtue; it is essential for readers to assess the meaning of p-values.
A five-column decision table from data to action
Each primary result can be summarized in five columns: point estimate; 95% interval; minimal meaningful difference; primary costs and risks; next step. Example 1 might read: “+1.0 percentage points, 0.62 to 1.38, threshold +3, high migration cost, do not launch.” Example 2: “+8.0, −0.7 to 16.7, threshold +3, direction uncertain, expand sample per pre-planned design.” If action is reversible and cost is low, even weak evidence may justify a small-scale trial; if it involves medicine, finance, or safety, stronger designs, external validation, and domain review are typically needed.
Before computing, set up an analysis plan and sample size, use confidence intervals to check precision, and identify population composition and multiple-comparison risks via cross-tabulation analysis. Final reports should put “what was observed, what is still compatible, and what threshold triggers action” together, rather than letting a single asterisk carry all of the interpretation.