A subscription experience study deliberately recruited 300 recent cancellers—who make up only 5% of the population—so that reasons for cancellation could be clearly seen. In the final sample of 1,000, 56% said billing was clear; after reweighting to match the customer population structure, the result was 72%. The 16-percentage-point gap between the raw and weighted values is not a calculation error. It arises because this sample was asked to do two opposite jobs at once: make a small group large enough to analyze, and estimate all customers.
Quotas are not simply “fill in demographic cells and representativeness follows.” Every quota must be labeled with its purpose: one type keeps the sample close to the target population, the other ensures sufficient analytical precision for key subgroups. Natural proportions suit overall description; equal-size or oversampled quotas suit comparison; if you oversample and then wish to recover the population, you need credible population shares, proper weights, and variance calculations that reflect the design.
Decide First: Protect the Overall Population or Prioritize a Small Key Subgroup
| Goal | Typical Quota | Strengths | Costs |
|---|---|---|---|
| Estimate the market or customer population | Natural allocation or close to natural proportions based on a credible population structure | Smaller weight variation, higher overall efficiency | Rare groups may end up with very few cases |
| Compare key subgroups | Equal size, minimum base, or oversampling | Small subgroups have usable precision | Unweighted overall picture is distorted; weighted variance increases |
| Experimental comparison | Random equal-size assignment within eligible samples | Greater efficiency for between-group comparisons | Quotas do not replace random assignment |
| Control source risk | Limit share from any single vendor or channel | Prevents sample domination by one access point | Source shares are not population shares |
If a study needs both population estimates and an understanding of cancellers, you do not have to choose. Oversample cancellers, use each group's actual base for within-group analysis, and apply design weights for overall population estimates. But the two outputs must be specified before fieldwork begins—you should not decide only after seeing unflattering raw results to report the weighted numbers instead.
Marginal Quotas Can All Fill 100% While the Cross-Structure Is Completely Wrong
Suppose the target population is 50% female and 50% male, and on age, 50% young and 50% old. A sample of 500 young women and 500 older men satisfies both marginal quotas but contains not a single young man or older woman. If the combination of sex and age affects your key outcome, two green progress bars will hide serious empty cells.
Cross-classified quotas can control combinations, but the number of cells grows quickly. Five age groups × four regions × three customer states × two sexes creates 120 cells. With a total sample of 1,200, the average cell has only 10 people; in practice, rare cross-cells will be smaller, more expensive to fill, and more likely to be completed by special sources near the end. Rules for deciding what to control:
- Set cross-quotas for combinations that directly drive your conclusions and for which a reliable joint population distribution exists.
- Control marginal quota only for variables that mainly maintain overall balance, but check key cross-tabulations at the end.
- Set minimum usable bases—not mechanical proportions—for groups that are small but need to be compared.
- For variables with no population benchmark and unclear links to outcomes, monitor first; do not impose hard quotas prematurely.
- Combine tiny cells that have no independent decision value and cannot be recruited stably—then document the limits of extrapolation.
The UK Government Social Research quota sampling guidance likewise requires design evaluation across study-relevant variables, population information, interlocking or cross-quotas, sample sources, fieldwork execution, weighting, and reporting. The quota table is only one element.
The 1,000-Sample Case: To See the 5% Cancellers Clearly, the Unweighted Population Estimate Dropped by 16 Points
The following is illustrative fictional data—not a representation of Wenjuan Pai customers, any subscription business, or industry benchmarks. The target population is customers who made a renewal, pause, or cancellation decision in the past year. A reliable business ledger shows 70% active renewers, 25% paused or low-frequency users, and 5% recent cancellers. Because the study must compare cancellers, completion quotas are set at 400, 300, and 300:
| Customer Status | Population Share | Completes | Said Billing Is Clear | Within-Group Positive Rate |
|---|---|---|---|---|
| Active renewers | 70% | 400 | 320 | 80% |
| Paused or low-frequency | 25% | 300 | 180 | 60% |
| Recent cancellers | 5% | 300 | 60 | 20% |
Unweighted, the positive rate is (320 + 180 + 60) ÷ 1,000 = 56%. Yet cancellers make up 30% of the sample—six times their population share—while active renewers represent only 40%, well below the 70% in the population. If the goal is an all-customer estimate, the calculation should follow population shares:
70% × 80% + 25% × 60% + 5% × 20% = 72%
The 72% answers “what about all customers”; the within-group rates of 80%, 60%, and 20% answer “how do customer states differ.” With 300 cancellers, the random uncertainty around their rate is far smaller than the roughly 50 cases you would have under natural proportions. That is the value of oversampling. It does not make cancellers more numerous in the real population.
Weights Can Restore a Known Population Structure—and Also Reduce Effective Information
Compute base post-stratification weights from each group's sample share: active renewers are 0.70 ÷ 0.40 = 1.75; paused or low-frequency users are 0.25 ÷ 0.30 ≈ 0.833; recent cancellers are 0.05 ÷ 0.30 ≈ 0.167. After weighting, the three groups sum to 700, 250, and 50, totaling 1,000, and the weighted overall positive rate is 72%.
Weights are not free, however. Under the common Kish approximation, this set of weights yields a sum of squares of about 1,441.7, with a weight sum of 1,000:
Effective sample size ≈ (Σw)² ÷ Σw² = 1000² ÷ 1441.7 ≈ 694
That means 1,000 nominal completes used for a weighted population estimate carry roughly the information of an equal-weight simple random sample of about 694. The weighting design effect is about 1.44. Pew Research Center's weighting diagnostics likewise use design effect and effective sample size to reveal precision loss from weight variance. For population reporting, use statistical methods that accommodate weights and complex designs; when reporting the cancellers' own 20%, state that the unweighted base consists of 300 actual responses.
Trimming extreme weights can buy back some precision but leaves more bias; you should not trim arbitrarily to obtain a prettier effective sample size. Instead, set limits a priori, compare key estimates before and after trimming, and explain the trade-off. If population shares come from ledgers that are outdated, use a different definition, or under-cover the target group, precision weighting merely aligns precisely to the wrong target.
Quota Variables Must Pass Three Gates: Relevant, Reliable, and Feasible
Relevant to coverage or key outcomes. Age, region, and customer state may matter because access channel, behavior, or outcomes vary with them; variables that merely “look professional” should not enter automatically. If a variable only affects report layout, describe it after data collection.
Have a population benchmark on the same definition. The benchmark must list its source, date, target population, category definitions, and treatment of missing data. A national adult population distribution cannot directly serve as the distribution of buyers in the past 90 days for a specific category, and registered-account structure is not the same as actual-user structure.
Can be screened accurately and recruited steadily. The quota field needs a clear question or frame variable—not vague self-reports such as “high spender” or “trendy.” If a cell requires different languages, offline outreach, or higher incentives, build that into the budget and timeline rather than improvising at the end of fieldwork.
| Quota Variable | Purpose Label | Benchmark and Version | Control Method | Contingency If Target Not Met |
|---|---|---|---|---|
| Customer status | Key comparison + population restoration | Quarterly customer ledger, same reference period | Minimum 300 cancellers; soft targets for others | Extend the field period; do not change status definitions |
| Region | Coverage balance | Regional distribution of eligible customers | Marginal tolerance range | Add sources, then re-check channel differences |
| Recruitment source | Risk control | No population target | Set source cap and retain source variable | Run source-sensitivity analysis |
The “Last Ten Completes” of a Hard Quota Are Often the Riskiest
The easy cells close in the first two days; the rare cells are left until the final day, when you switch vendors, raise incentives, or loosen screening criteria. At that point population, source, timing, and incentive shift simultaneously, so the last cases may be systematically different from earlier ones. The quota is achieved, but the measurement is no longer consistent.
Field management should define in advance: whether a quota is a hard cap, a soft target, or a minimum base; how much deviation is allowed; at what point source expansion is triggered; whether over-quota completes are retained; who has authority to change quotas; and which results require stage-by-stage comparison after any change. Every modification should be recorded with effective time, reason, channel, invitations sent, and affected responses. Never change quotas simply because preliminary results are not what you expected.
Quality rules must also apply consistently across cells. A hard-to-recruit group is not an excuse to relax logic checks; an easy one should not be cleaned more aggressively. If cancellers more often give negative open-ended answers, do not classify that as “emotional” low quality. First check completion paths, attention, contradictions, duplicates, and response patterns; then apply rules that do not depend on knowing the direction of your main conclusions.
Nonprobability Samples Quota-Filled to Population Cells Are Still Not Probability Representative
In stratified probability sampling, selection probabilities per stratum come from the design, and oversampling can be undone with inverse-probability weights. In a volunteer opt-in online sample or open link, the probability of seeing the invitation, joining the platform, and agreeing to respond is usually unknown. Quotas only make the observed age, sex, or region distribution resemble the target; they do not show that respondents and nonrespondents inside a cell share the same attitudes and behaviors.
AAPOR's report on nonprobability online samples clearly distinguishes design-based from model-based inference: without known inclusion probabilities, you cannot rely on the mathematical properties of probability survey sampling. Companies can still use quotas for directional research, comparing options, or exploring mechanisms—but they should disclose their sources and limits rather than describing “complete quotas” as “nationally representative random sampling.”
In Wenjuan Pai, the Quota Table Should Also Be a Decision Ledger
Wenjuan Pai can control recruitment in real time by screening questions, customer status, and region, while retaining source for later analysis. Before configuration, write down for each quota its purpose, benchmark, target, minimum base, tolerance, weighting plan, and fallback route, so that research, sample operations, and analysis all use the same definitions.
A possible published result for this case would read: “To compare recent cancellers, the study oversampled them from 5% of the population to 30% of the sample. The 56% figure is a description of the unweighted sample composition and does not represent all customers. After weighting by customer-state shares for the same period, the overall billing-clarity estimate is 72%, with an approximate weighted effective sample size of 694. Within-group results are 80% for active renewers, 60% for paused or low-frequency users, and 20% for recent cancellers, based on actual unweighted bases of 400, 300, and 300, respectively.”
This statement preserves both the population conclusion and the value of studying a small group. Good quota design is not about every cell being perfectly filled. It is about the reader being able to tell which number serves which decision, how much precision the weights cost, and which unobserved differences still cannot be adjusted away.
As next steps, you can combine defining your target survey audience to lock down the population, use sample size calculation to set key cell bases, and read survey weighting basics to plan your estimation approach.
Evidence Base for Quotas, Oversampling, and Weighting Efficiency
- UK Government Social Research — Quota Sampling Guidance
- U.S. Census Bureau — Statistical Quality Standard A3: Sample Design
- United Nations Statistics Division — Household Sample Surveys and Design Effects
- Pew Research Center — Weighting survey data and effective sample size
- AAPOR — Nonprobability Online Survey Samples