In a customer support questionnaire, a nine-row, five-column matrix appears. Column headers range from “Very Dissatisfied” to “Very Satisfied,” yet the rows mix “waiting time,” “issue resolved,” “agent professionalism,” “willingness to use again,” and “frequency of self-service center use.” The page looks neat and the data are all 1–5, but these numbers carry no shared meaning: outcome facts are forced into satisfaction ratings, frequency is mistaken for attitude, and future intention is mixed in with the current experience.
A matrix is merely a layout that places multiple items on one page; it is not proof of measurement quality. Sharing a set of column responses across rows does not mean the rows measure the same construct, let alone that averaging them is appropriate. The design order should be “research construct → item wording and response scale → target population → page presentation,” not picking a nice-looking matrix first and then squeezing every conceivable content into row labels.
First, distinguish three frequently confused objects
| Object | What it is | Primary judgment |
|---|---|---|
| Matrix/grid | Multiple row items share column options on the screen | Readability and operability on the target device; whether items are at the same level of abstraction |
| Rating response item | One question uses ordered categories from low to high | Whether the question stem and anchor points match; whether respondents can reliably distinguish the levels |
| Multi-item construct scale | Multiple theoretically and empirically supported items jointly measure a latent concept | Whether item content, dimensions, scoring, reliability/validity, and target population hold |
A five-point single item does not automatically become a reusable psychometric scale simply because it uses “strongly disagree—strongly agree”; nor does a nine-item table constitute a composite index merely because statistical software can compute an average. If the goal is to examine nine service touchpoints separately, report each item’s distribution individually. If the goal is to create a “service trust” score, you must demonstrate at the design stage which items belong to that construct, how they are scored, and for which populations they apply.
How to break the old nine-row matrix into four explainable data types
First, ask respondents to recall their most recent completed customer service event and confirm which channels they used. Then classify each old matrix row:
| Old matrix content | True variable type | Rewriting approach |
|---|---|---|
| Whether used online chat, phone, or help center | Experience/exposure | Multiple-choice factual item; only actual channels used are carried forward |
| Whether issue was resolved | Outcome status | Yes, partially resolved, not resolved, unsure |
| Finding the entry, understanding explanations, completing actions | Task difficulty | Very difficult, somewhat difficult, neither easy nor difficult, somewhat easy, very easy |
| Waiting and processing speed | Time experience | Record a recallable time interval, then separately ask about speed satisfaction (if needed for decisions) |
| Agent professionalism and communication | Personnel experience | Show only to those who interacted with a human agent; can form a short battery under the same satisfaction stem |
| Willingness to use again | Future intention | Ask separately after event evaluation; do not average into the current service score |
This step increases the number of pages but reduces meaningless numbers. People who never used the phone should not be forced to rate phone satisfaction; “issue not resolved” is not a synonym for a satisfaction score of 1. Experience determines whether an item applies; outcome, process evaluation, and future intention answer different decisions.
The same column header must complete the same sentence for every row
To determine whether a matrix is valid, read the question stem and each row label together. For example, “Regarding your most recent customer service event, how easy or difficult was it to complete the following tasks? — Finding the contact entry / Describing the problem / Understanding the solution.” All three rows naturally follow the same stem and use the same difficulty scale.
If you read “How satisfied are you with ‘issue resolved’?” that recasts an objective outcome as an attitude; “How satisfied are you with ‘frequency of using customer service’?” is semantically unsound. A qualified matrix meets at least the following:
- Row items are at the same level of abstraction—no mixing of brands, touchpoints, attributes, and outcomes;
- Each row uses the same subject, reference event, and time frame;
- All respondents are capable of evaluating, or logic exposes only applicable rows based on their experience;
- The same number is directionally consistent and means the same thing across rows;
- Each row is a variable genuinely needed for analysis or action.
Five points versus seven: no task-free standard answer
The number of scale points should depend on whether respondents can distinguish, the resolution the construct requires, the administration mode, sample size, and historical comparability. Five points are usually shorter and easier to label fully; seven points offer more gradations, but the extra categories are information only if the target population can use them reliably—otherwise they are decoration. For validated scales or trend indicators, preserve the original design; do not casually change the number of points for page aesthetics.
More critical than the number of points is semantics:
- If the stem asks satisfaction, use satisfied—dissatisfied; if asking difficulty, use difficult—easy; do not mix;
- The end anchors should be symmetric; provide clear labels for each category where possible, not just 1 and 5;
- The midpoint “neither satisfied nor dissatisfied” is a substantive position, not “don’t know”;
- “No experience/not applicable,” “don’t know,” and “refused” should fall outside the scale, with separate codes;
- Keep the direction of similar scales consistent throughout the questionnaire—do not have high scores meaning good on one page and bad on the next.
The UK Government Analysis Function’s questionnaire design guidance emphasizes that response categories must match the question, avoid overlap, control the number, and keep direction consistent across similar scales. It provides five- and seven-point examples, but “examples” do not mean all constructs must adopt the same number of points.
Don’t set language traps for careful respondents with reverse-coded items
Double negatives like “The agent did not fail to earnestly solve my problem” may seem to catch straightlining, but they also risk confusing careful respondents, especially in cross-language or small-screen environments. If a theoretical scale includes positive and negative items, use them per the validated protocol and reverse-score correctly in analysis. If you improvise negative statements only to detect cheating, the observed differences may reflect language-processing ability rather than the target attitude.
When you need to check acquiescence, use items that are semantically clear yet differ in content direction, and combine cognitive testing with inter-item relationships. Quality control should not be achieved by deliberately making items harder.
On mobile, present items one at a time by default; a matrix requires extra justification
Traditional matrices on mobile often fail to display row labels, column anchors, and selection controls simultaneously; after horizontal scrolling, respondents may forget the meaning of the current column. The U.S. Census Bureau’s mobile survey design standards proposal recommends replacing traditional grids with item-by-item questions and summarizes evidence that splitting can reduce mobile straightlining and item nonresponse while improving cross-device comparability.
Pew Research Center’s mobile web survey tips also note that grids can encourage same-column answering and introduce vertical and horizontal scrolling, misrecording, and dropout on phones. Item-by-item presentation does not necessarily require loading a new page per question: you can stack a small number of items vertically on the same page with full options visible next to each item, while avoiding frequent network page jumps.
The 2026-updated U.S. Census Bureau web questionnaire design guidelines place guidance for item types like grids within an evidence spectrum comprising randomized experiments, usability testing, heuristics, and practical evidence. This means layout choices should be validated on real devices and target populations—not concluded from a single desktop preview.
Use a randomized usability experiment with 120 participants to compare layouts
Suppose the nine rows have been condensed into six homogeneous “task difficulty” items, but the team is still unsure whether a traditional matrix or item-by-item stacking is better. Recruit 120 participants similar to the target population outside the production sample, stratify by mobile/desktop, and randomly assign them to the two layouts, 60 per group. The two versions are identical in stem, items, order, and response options; only the presentation changes.
Pre-register before viewing results:
- Primary usability outcomes: whether participants completed the six items without help, and how many needed to return and modify answers;
- Data outcomes: item nonresponse, option misclicks, same-column responding, and survey breakoff;
- Process outcomes: time on the item set, horizontal scrolling, zooming, error recovery, and device;
- Comprehension outcomes: after completion, ask “What does level 3 mean?” and “Which experience were you rating just now?”;
- Decision rule: if the matrix saves time but is accompanied by more errors or cross-device differences, adopt the item-by-item layout.
For example, if the experiment observes a median completion time of 52 seconds for the traditional matrix, with 9 people returning to fix responses and 6 missing at least one item, versus 65 seconds for the item-by-item version with 3 fixing and 2 missing, you cannot declare the matrix the winner simply because it is 13 seconds faster. Assess errors, comprehension, and mobile differences; these numbers are illustrative experimental results for this project, not industry benchmarks to generalize.
Selecting the same column for all rows is not automatically invalid
A participant who selects “fairly easy” for all six items, takes 55 seconds, rates items they actually experienced, and provides a specific process description in an open-ended answer may be reporting a genuinely consistent experience. Another who selects the midpoint for all six items in seven seconds, including two items they had never experienced per prior questions, and gives irrelevant open-ended comments, is a case of joint evidence—“extremely short duration plus not-applicable contradictions plus low-relevance text”—rather than straightlining itself.
Continuing the illustrative experiment: suppose 23 of 60 participants in the traditional matrix gave same-column responses. Upon review, 14 had plausible experience, timing, and text, so keep them; 6 simultaneously triggered multiple pre-registered low-quality rules and can be flagged for exclusion; the remaining 3 show only weak signals, and no human should subjectively guess their motivation. When reporting quality processing, retain raw rows, signals, and rule versions.
The straightlining rate also depends on content: if the six items are naturally similar, consistent answers are not surprising. Inserting irrelevant or reverse-coded items to force variety damages construct validity. Better prevention is to reduce item count, ensure each row has discriminating value, show rows only to those with relevant experience, and make the mobile presentation readable.
In analysis, first look at distributions, base sizes, and applicability
For every item, report at minimum the valid base, full category distribution, not-applicable and item nonresponse. A mean can serve as a summary, but whether the distances between ordered categories are approximately equal is an analytic assumption; do not present only a “4.2 score” while hiding a large share clustered at the ends. When comparing groups, verify that the scale translation, presentation mode, and item meaning are consistent.
Only form a total score when multiple items are actually designed to measure the same construct and have supporting dimensionality and scoring evidence. High internal consistency alone does not prove unidimensionality or validity: several near-paraphrased items can be highly correlated yet simply repeat one narrow wording. Scoring rules, minimum answered items, missing-data handling, and reverse-scoring should be written into the analysis plan in advance.
In Wenjuanpai, item-type selection follows measurement specifications
Teams can create rating items or matrices in Wenjuanpai and use logic to show only items relevant to a respondent’s experience. After configuration, check on target phones: question stem, full labels, selection state, return-and-modify, and exported variables. If randomizing rows without a natural order is needed, save the actual presentation order; see survey randomization design for methods.
Evidence that a matrix is publishable is not “nine rows happen to fit on the page” but that each row connects to the same question stem, uses the same construct and response scale, all raters possess the relevant experience, mobile experiments show the layout does not increase meaningful errors, and analysts know which items can be reported separately versus which require validation before aggregation. A tidy page is merely a byproduct; semantic equivalence is the prerequisite.