How to Design Questionnaire Randomization? First Distinguish Random Order, Randomized Experiments, and Random Sampling

Three beverage concepts were always shown as A, B, C, and A had the highest purchase consideration. The team planned to keep A, but then noticed A was always shown first: it may have benefited from the freshness of full attention, while C suffered from comparison and fatigue. In the next round, they simply turned on “randomization,” and although the order of the three concepts varied, the export did not save the actual order each respondent saw. The results remained uninterpretable.

Questionnaire randomization is not a universal switch. Designers must first answer: Are you trying to spread out the influence of list position, estimate the causal difference between different phrasings or stimuli, or draw an inferential sample from a population? These three uses of “random” involve different randomization units, data fields, and analytical conclusions. Random presentation does not automatically produce a representative sample, and random sampling does not automatically eliminate question order effects.

The Three “Randoms” Answer Three Different Questions

There is a fourth common operation: randomly selecting four out of twelve features for each respondent to rate. This reduces individual burden but changes the analysis base for each feature from the total sample to the selected subsample; it is not equivalent to merely changing the display order of the twelve features.

Before Randomizing, Write Down the Confounders to Control

Randomization has costs: it adds versions, test paths, and analytical complexity. Therefore, don't randomize everything just because the platform offers the feature. First, write down “Object – Confounder – Randomization Unit – Fixed Items – Saved Fields – Primary Comparison” in a design table. For example:

• Brand awareness multiple-choice list: Concerned items at the top of the webpage are more visible; randomize the position of non-anchor brand options;
• Three full concepts: Concerned about the freshness effect for the first viewer and fatigue for later viewers; randomize the order of the concept blocks;
• Price framing experiment: Want to estimate the impact of “monthly” vs. “daily equivalent” framing; randomly assign respondents to one version;
• Core indicators in a tracking survey: Need to maintain the same context as the previous wave; do not randomize key question order unless incorporating the methodological change into a bridge design.

If confounders can be directly controlled, such as language, customer tier, or recruitment channel, consider stratifying/blocking first, then randomizing within strata. The NIST documentation on randomized block designs summarizes this as: block what you can, randomize the rest. While originally intended for experimental processes, the principle applies equally to survey experiments—to prevent one stimulus version from being assigned entirely to enterprise customers or a specific channel.

Which Options Can Be Shuffled, Which Must Be Pinned

Suitable for randomization are typically categories that are semantically parallel, lack a natural order, and where each item should get roughly equal exposure, such as brands, purchase channels, or independent product attributes. The following items generally should not be shuffled with regular list items:

• Ordered categories like age groups, income brackets, frequency, or agreement levels;
• Special statuses like “Other, please specify,” “None of the above,” “Don’t know,” or “Refused”;
• Funnel questions moving from broad to specific, and sequences where later questions depend on definitions from earlier ones;
• Timelines, process steps, or stimuli that must be understood in a specific reading order;
• Core indicators with a historically fixed order used for trend comparison, unless explicitly evaluating a break in series.

Pew Research Center’s guide on writing survey questions notes that primacy effects are common in web self-administered lists, while recency effects may occur in telephone interviews; randomization or rotation does not eliminate order effects but spreads them across items. Ordered scales typically maintain their order because the category positions themselves convey a cognitive structure from low to high.

Randomizing options also cannot fix an incomplete list. If key answers are entirely missing, shuffling them evenly only produces errors evenly. First ensure options are exhaustive, mutually exclusive, and use the target population’s language, then address the issue of position.

Full Rotation of Three Concepts: How to Allocate 720 People to Six Orders

Returning to the A, B, C beverage concepts. If everyone must rate all three and rate them separately, all possible orders are:

This way, each concept appears 240 times in the first, second, and third positions, and each pair order (e.g., A before B) is also balanced. The actual number randomly assigned doesn't have to be mechanically 120; if the research requires precise balance, pre-generated random sequence blocks can be used, but field staff must not be allowed to choose the next sequence based on preference.

For analysis, at a minimum, retain the concept, position, previous concept, full sequence, and rating. First look at the main effect of the concept, then check for an interaction between position and concept. If C is only significantly lower when in the third position, the report should disclose the fatigue or comparative context rather than mixing all orders and claiming C is inherently worse.

Within-Subjects vs. Between-Subjects Designs: Neither is Universally Superior

Within-subjects design: The same person rates A, B, and C. The advantage is that each person acts as their own benchmark, and all concepts get ratings from all 720 respondents. The disadvantage is that the task is longer, earlier stimuli can influence responses to later ones, and respondents might guess the study’s purpose.

Between-subjects design: Randomly divide the 720 respondents into three groups; each person sees only one concept, providing approximately 240 ratings per concept. This reduces fatigue and cross-stimulus contamination and more closely mimics independent exposure. However, the base size for each concept is smaller, and individual differences between groups add more uncertainty. If the ultimate market scenario involves consumers seeing only one full proposition, a between-subjects design is often more realistic; if decisions require the same target customer to make relative trade-offs with short stimuli, a within-subjects design might be more efficient.

Do not collect data for A, then for B and C on subsequent days after finishing the first group by channel. Even if group sizes are equal, date, channel composition, or market events would be completely confounded with the concept. Random allocation should occur after eligibility confirmation, before outcome measurement, keeping recruitment concurrent.

Why Randomized Experiments Support Causal Interpretation

If eligible respondents are assigned to Text X or Y in an unpredictable way during the same period, and the process is otherwise identical, then with a sufficiently large sample, the two groups have the opportunity to be comparable on both measured and unmeasured background characteristics. Differences in outcomes between the groups can then be attributed to the manipulated content, with an associated uncertainty interval. The World Bank’s “Impact Evaluation in Practice” lists random assignment as one method for constructing treatment and counterfactual comparison groups.

This still has boundaries. Randomization assigns respondents who entered the study; it does not turn a convenience sample from a public link into a probability sample of all consumers. The results can only robustly indicate the average difference for this study population under these materials and measurement conditions. If someone bypasses assignment, reopens the survey until they see the desired version, or if different groups use different channels, the causal interpretation is compromised.

Pew’s explanation of survey experiments shows how randomly varying question wording, response options, and question order helps identify presentation effects. The key is not merely that “two versions appeared on the page” but that respondents were randomly assigned, other conditions were comparable, and the analysis did not cherry-pick favorable versions after seeing the results.

When Randomly Selecting Four Features, Design the Sample Based on Item Base Sizes

If there are 12 features and each respondent randomly selects 4 without replacement from a total sample of 720, the expected number of ratings for each feature is:

720 × 4 ÷ 12 = 240

This is only the expectation, not a guarantee that each item gets exactly 240 ratings. Researchers should plan precision based on approximately 240, not 720, and monitor actual base sizes. If three items are core and required, display those three fixed, and randomly select 1 from the remaining nine. In this case, core and non-core items have different inclusion probabilities, which must be documented in the data dictionary; raw response counts cannot be directly compared.

Sampling also changes co-occurrence information: if two items are rarely shown together, it becomes difficult to estimate their correlation or preference difference within the same respondent. If trade-off combinations are needed, co-occurrence probabilities should be designed, not just individual item counts.

Stratification, Clustering, and Repeat Entry Must Be Decided Before Assignment

In small-sample experiments, stratify by the most important, pre-known variables such as customer type and language, allocating X and Y roughly equally within each stratum. Over-stratifying creates many small cells, making the rules fragile; only select a few variables strongly related to the primary outcome or that are critical for business balance.

When team members are likely to discuss materials with each other, randomizing by individual can cause contamination. Consider randomizing by team, but acknowledge in the analysis that the effective randomization unit is the team, not the individual; the number of clusters needed often becomes the binding constraint. Repeat entries should maintain the original allocation for the same person; re-randomizing on every page refresh is not allowed. The identifiers used for persistence and associated privacy handling need to be determined during the design phase.

Without These Fields, Randomization Cannot Be Audited

• randomization_scheme_version: Which version of rules and stimuli were used;
• assignment_unit_id: Whether it was an individual, session, customer ID, or group;
• assigned_arm and actual_exposure: What was assigned vs. what was actually displayed;
• display_order and subset_ids: The full position and the selected subset items, not just the final answers;
• assigned_at, eligibility status, stratification variables, and recruitment source;
• Any manual overrides, reassignments, loading failures, and their reasons.

Primary analysis typically compares groups based on initial assignment to preserve the randomized design; actual exposure fields are used to diagnose implementation deviations and should not be used to arbitrarily exclude people who didn't see the full material when the group effects are unfavorable. Detailed inclusion, exclusion, primary endpoints, and interaction tests should be registered in a pre-collection analysis plan.

Don’t Use “50/50 Looks Close Enough” as a Pre-launch Check

First, use test sessions to verify each version is accessible, fixed items are not shuffled, dependent questions always follow their source questions, materials and export identifiers match, and back/return or resume functions do not trigger re-randomization. Then, use a sufficiently large number of simulations or pilot allocations to check:

• No version receives zero or abnormally few assignments;
• Allocation rules within each stratum match the specification;
• Position and pair-order combinations of concepts achieve the designed coverage;
• Channel, date, and device are not deterministically tied to one version;
• Each experimental arm has independent and consistent completion, quality control, and export processes.

A truly random small sample showing 47/53 does not automatically indicate a programming error, and forcing an exact 50/50 split for every ten people might also reveal a predictable assignment pattern. Validation should check the mechanism and long-term coverage, while defining acceptable imbalances and stopping rules in advance.

In Wenjuanpai, Configure the Design First, Then Verify Assignment Records

When using Wenjuanpai to configure the random display of options, blocks, or stimuli, clarify the randomization unit, fixed items, sample count, and stratification specs first, then use preview and test responses to verify the actual order and the export. The platform executes the randomization rules, but the researcher must decide which question the rule answers, how many valid responses are needed per version, and whether data from old versions can be merged after mid-field changes.

A randomization plan is truly complete when the team can reconstruct any row of responses: why this person was eligible, when they were assigned to which arm under which rule version, what materials they actually saw in what order, and which results entered the primary analysis. If you can only see the final ratings without knowing the display history, the randomization toggles may be on, but the study design remains unauditable.

Basis for Order Effects and Random Allocation

• Pew Research Center: Writing Survey Questions
• Pew Research Center: Survey Experiments
• NIST/SEMATECH: Randomized Block Designs
• World Bank: Impact Evaluation in Practice