How Many Responses Do You Need? 400 Is Not a Universal Answer

Asking “is 400 enough?” omits the most critical qualifier. Using simple random sampling to estimate a population proportion near 50% with a target 95% confidence interval of about ±5 percentage points yields a required n of 385; but to detect a difference between 50% and 60% with 80% power, you need roughly 388 per group, or 776 total. If you want to detect a 5-point difference instead, you would need about 1,565 per group.

Sample size is not dictated by an industry rule of thumb but is jointly determined by the primary estimate, allowable uncertainty, the minimum meaningful difference, key subgroups, and the sampling design. Increasing the population from 100,000 to 10 million usually does not require a proportionate increase in sample size; however, splitting one population into four regions that must each be reported separately, using cluster recruitment, or needing to detect smaller differences will substantially increase the required effective sample size.

First Choose a Task: Estimating a Number or Comparing Two Numbers?

Before calculating sample size, lock down the analysis question. Four common tasks should not share the same calculator:

“What is the overall satisfaction rate?” is an estimation task; “Is the new version at least 10 percentage points better than the old one?” is a comparison task; “Which factors predict churn?” is a modeling task. Select one primary outcome and one primary analysis in advance, and treat other metrics as secondary outcomes, to avoid back-explaining the sample size with the most conveniently significant result.

Estimating a Proportion: Where 385 Comes From

Under the basic scenario of a large population, simple random sampling, approximately independent observations, and use of the normal approximation, the sample size for estimating a proportion is:

n₀ = z² × p × (1-p) ÷ e²

z is the critical value for the confidence level (approximately 1.96 for a two-sided 95% interval); p is the expected proportion at the planning stage; and e is the allowable half-width, i.e., the familiar “± percentage points.” In the absence of a reliable prior, set p=0.5 because p(1-p) is maximal there, giving a more conservative sample size.

Plugging in p=0.5 and e=0.05:

n₀ = 1.96² × 0.5 × 0.5 ÷ 0.05² = 384.16

Since sample size must be rounded up, this yields 385 completed responses. If credible historical data suggest a proportion around 20%, with the same allowable error of ±5 points, the approximation is 1.96²×0.2×0.8÷0.05²=245.86, rounded up to 246. If your expected proportion is wrong, the actual interval could be wider than planned, so you should not arbitrarily pick a value far below 50% merely to cut costs.

The NIST documentation for sample sizes for proportions similarly requires specifying the confidence level, allowable sampling error, and population proportion, and treats p=0.5 as the most conservative case when the true proportion is unknown. The “±5 percentage points” here describes only the random error under that sampling model; it does not include non-coverage, nonresponse, question misinterpretation, cheating, quota deviations, or processing errors.

With a Population of Only 2,000, the Finite Population Correction Is Noticeable

When sampling without replacement a substantial fraction from a fixed, finite list, you can apply the finite population correction:

n = n₀ ÷ [1 + (n₀-1) ÷ N]

If the required n₀ is 384.16 and the population is N=2,000, the corrected sample size is approximately 322.4, rounded up to 323. If N=100,000, the result is about 382.7, still near 383. This shows that expanding the population from 100,000 to 10 million is not a reason to multiply the sample size; what matters is the planned precision, the subgroups, and the design.

The correction applies only if those 2,000 people truly constitute the complete target population and can be sampled with known probability. If the client list misses off-line users, contains duplicates, or includes many people who cannot be reached, the mathematical correction will not fix coverage bias. If a substantial portion of the list is expected not to respond, you must overdraw beyond the 323 completed samples.

A Sufficient Total Is Not Sufficient for Each Key Subgroup

If the team must report separate proportions for the East, South, West, and North regions, each needing precision of about ±5 points when the proportion is near 50%, the target is not 385 in total but about 385 per region—roughly 1,540 in total. If you simply divide the total 385 equally across four regions, each would have about 96, yielding an error of nearly ±10 points under the same simple approximation.

You do not need to guarantee the same precision for every cross-tabulation. Before data collection, list the subgroups that “must drive decisions,” such as region and new vs. existing customers. Exploratory cuts like age × region × channel can be collapsed, reported with wider intervals, or withheld altogether. For rare but decision-critical groups, consider oversampling and then weight back during analysis. Plan the sample size around the largest core requirement, rather than discovering after collection that one cell has only a dozen people.

Version Comparison: The Smaller the Difference, the Faster the Sample Grows

Suppose the old version’s completion confidence is 50% and the team decides a lift to 60% is needed before full rollout, with a two-sided significance level of 5%, statistical power of 80%, and equal random allocation between groups. Under the common normal approximation for two proportions, you need about 388 per group. This being close to the 385 for a single proportion is a coincidence—the calculation goals are entirely different.

If the business decides that 55% is also worth launching, the target difference halves from 10 to 5 points. The sample size does not double; it jumps to about 1,565 per group. Because the standard error shrinks only with the square root of the sample size, halving the detectable difference typically requires nearly quadruple the sample. Conversely, if the budget supports only 200 per group, you should compute the detectable difference or present a range of plausible results, rather than writing off an underpowered “not significant” as “the two versions are the same.”

The DELTA² guidance emphasizes that the target difference in comparative research should be justified as realistic and important before the study, and reported alongside the primary outcome, significance level, and power. The final calculation also depends on whether the test is one- or two-sided, whether the outcome is continuous or binary, the baseline rate, allocation ratio, repeated measures, covariates, and the chosen method. So use validated software for confirmation rather than relying only on the approximations presented here.

600 Nominal Responses May Carry Only 480 Units of Information

The simple random sampling formula treats each response as contributing similar information. In practice, clustering, unequal selection probabilities, and substantial weights can alter the variance. The WHO defines the effective sample size as the sample size that would be needed under simple random sampling to achieve the same uncertainty; for a complex design, the effective sample size is approximately the actual sample size divided by the design effect.

Suppose that among 600 respondents, half have a weight of 0.5 and half a weight of 1.5. The weights sum to 600 and the sum of squared weights is 750. Using the common weight-based approximation for effective sample size:

n_eff = (Σw)² ÷ Σw² = 600² ÷ 750 = 480

This corresponds to a weight-related design effect of 1.25. Under this approximation, 600 completed interviews provide the information of roughly 480 equally weighted independent samples. While weighting corrects known structural biases, it also increases variance. Extreme weights should be addressed during design and fieldwork, not simply computed away afterward.

Clustering also lowers effective information. If each store contributes about 20 respondents on average and the intraclass correlation of responses within a store is estimated at 0.05, the approximate design effect is 1+(20-1)×0.05=1.95. To obtain the information equivalent to 385 simple random samples, the nominal number of completes would need to be about 751. Adding more stores is usually more efficient than adding more respondents to existing ones, because the latter provide increasingly similar information. Stratification can sometimes improve precision, but this must be evaluated under the actual design; it cannot be assumed to offset clustering.

From Completed Responses Back to Invitations: Multiply Three Sources of Attrition

In the two-version comparison above, you need 776 completed responses. Suppose you invite from a broad population, and expect that 40% of invited people are eligible for the target, 75% of eligible respondents complete the survey, and 90% of completes pass quality checks. The probability that an invitation becomes a usable response is:

0.40 × 0.75 × 0.90 = 0.27

The required number of invitations is then 776 ÷ 0.27 = 2874.1, or at least about 2875. If quotas by region are also required, compute these per cell; the hardest quota may exhaust the contactable list first. Eligibility, completion, and pass rates should come from previous comparable studies or a soft launch, not from vague averages supplied by the vendor.

Quality loss cannot be handled casually with “let’s just add 20%.” Cleaning rules should be defined before data collection and monitored for accidental exclusion; if low quality concentrates on a particular device, language, or accessibility path, deletion will shift the sample composition. Expected loss is only a budget buffer—the final report should still give actual counts invited, screened out, incomplete, removed for quality, and available for analysis.

For Small Proportions, Models, and Longitudinal Studies, Use a Different Planning Unit

If an event occurs in only 2% of the population, a sample of 400 yields only about 8 expected events. Although the overall margin of error may look acceptable, you will not be able to stably compare event populations or fit models with several covariates. In such cases, start from “how many events or how many cases per category are needed,” then derive the total sample, oversampling where necessary and using appropriate weights to recover population estimates.

For longitudinal tracking, plan around the final wave’s required number of complete matched pairs. If 400 people are needed at the final wave and you expect 80% retention per wave, the start sample should be at least about 400 ÷ 0.8² = 625 across two waves. Different drop-out groups may have different experiences; increasing the start sample only preserves quantity, it does not automatically eliminate attrition bias. Design the study to include tracing efforts, compare those who stay with those who drop out, and state precisely whose results the final estimates apply to.

Qualitative interviews, usability tests, and cognitive interviews have their own sampling logic—covering key experiences, saturation of themes, and mechanism discovery. They should not be forced into a proportion-error formula. They can explain “why,” but small purposive samples cannot estimate population shares. Mixed-methods studies typically use qualitative material to sharpen measurement and refine mechanisms, then plan the sample size for the primary quantitative estimate.

Write the Sample Size as an Auditable Plan, Not a Procurement Number

Before the formal launch, keep a record of the sample size calculation: target population and sampling frame, analysis unit, primary outcome, planned estimate or test, baseline and variance sources, allowable error or minimum meaningful difference, confidence level, significance level, power, key subgroups, finite population correction, design effect, expected nonresponse and quality loss, final rounding, and the software version used. If inputs come from a small soft launch, state the uncertainty upfront and pre-specify how you will adjust.

Do not keep peeking at significance and topping up the sample until “something appears.” That changes the error rate. If mid-course looks, early stopping, or sequential decisions are needed, specify the appropriate sequential method and thresholds in advance. After the study, do not use “observed power” as a substitute for interpreting results. The STROBE guidance suggests stating how the study size was determined and presenting the actual precision using estimates and confidence intervals.

Field managers can monitor invitation, eligibility, completion, and quality-pass counts against the target population and quotas, updating feasibility assumptions after a soft launch. The statistical sample size is determined by the research design; the panel’s delivered sample is merely the implementation result. If respondents come from a non-probability opt-in channel, even reaching 2,875 or more does not allow you to use a probability-based ± error band to cover selection bias.

An approvable description could read as follows: “The primary comparison is a binary outcome with old version at 50% and new version at 60%, two-sided alpha = 5%, power = 80%, equal allocation. Approximately 388 completed responses per group are needed. Anticipating 40% eligibility, 75% completion, and 90% quality pass, we will invite at least 2,875 invitations. Regions will be reported only on an exploratory basis. If store-level clustering is introduced, sample sizes will be inflated based on the intraclass correlation from the soft launch.” When the numbers, assumptions, and decision boundaries coexist, the sample size is ready to be used.

For further planning, you may wish to review it together with the definitions of a target sample population, the sample quota strategy, and confidence interval interpretation.