A 12-row, 7-column satisfaction matrix that fits entirely on a 27-inch screen—on a phone, the question text stays fixed on the left while the last four scale columns remain hidden beyond the horizontal scroll. Respondents tap through the first visible three columns, and the system still receives a "complete response." The page doesn’t crash, but the data have already been shaped by the interface.
Mobile questionnaire design is not a by-product of responsive CSS; it is measurement design. Small screens, touch input, soft keyboards, network switching, and interruption scenarios change what participants see, what they must remember, what they can select, and whether they continue. The goal is not for the mobile page to be "openable" but for people on different devices to understand the same question, have the same response options, and be able to complete it with an acceptable burden.
First, Map Mobile Challenges to Four Types of Survey Error
| Manifestation | Actual Error Risk | Why Completion Metrics Alone Are Blind to It |
|---|---|---|
| Older devices or assistive technology cannot respond | Coverage error: a segment of the target population has no chance to participate | Backend only sees those who successfully entered |
| Concentrated drop-off on ranking questions or image-heavy screens | Unit nonresponse or nonresponse due to break-off | Completers still appear "high quality" |
| Keyboard obscures buttons, error messages unclear | Item nonresponse or erroneous values | Missing values can be misinterpreted as refusal or "don’t know" |
| Scale columns collapsed, labels hard to remember | Measurement error and device-related mode effects | Answers are in legal format, but meaning has shifted |
Therefore, when lower satisfaction is found on mobile devices, it cannot be directly interpreted as "mobile users are more dissatisfied." Mobile and desktop users themselves may differ, and the interface presentation may also alter responses. It is necessary to combine sampling, device-level paradata, usability observations, and, when necessary, randomize presentation or conduct bridge experiments to distinguish population differences from measurement differences.
The U.S. Census Bureau’s 2022 working paper on mobile survey instrument design connects mobile interfaces directly to measurement error: inappropriate interfaces can affect the perception of question information and the input of answers. The value of mobile-first thinking lies in starting from measurement risk, not visual aesthetics.
Calculate a "Mobile Task Budget" for Every Question
Total survey length is too coarse. Each question can be broken down into five types of burden:
- Perception: How many screens to scroll through; can the question text, units, and all answer options be seen at the time of answering;
- Memory: Whether respondents can still remember the scale endpoints, image differences, or preceding conditions when selecting an answer;
- Motor: How many taps, horizontal swipes, drags, zooms, keyboard switches, and corrections are required;
- Input: How much text, numbers, or dates need to be typed, and can the format be assisted by controls rather than after-the-fact error messages;
- Recovery: After an incoming call, screen lock, weak network, back button, or page refresh, can answers and position be reasonably preserved.
Tasks combining high perception with high motor effort are the riskiest. For example, asking respondents to compare four product images with small text, then drag them into full ranking, while also providing reasons, effectively piles visual discrimination, preference judgment, fine-motor manipulation, and textual expression into a single screen. First confirm that images are independently recognizable, then use touch-friendly selection methods, and finally follow up on only the most important choice with a specific reason.
Adapting a 12×7 Matrix to Mobile Is Not Just "Auto-Wrap"
The original task asked respondents to rate satisfaction with 12 functions on a 1-to-7 scale. Start with three research questions:
- Has every respondent actually used each of the 12 functions? If not, ask about usage first and only rate the items they have used.
- Can respondents reliably distinguish each of the 7 points? If the business only needs clear levels of satisfaction, reducing the scale may lower burden, but changes to an existing scale should consider trend comparability.
- Are all 12 items decision-relevant? Remove items with no owner, no action, or that duplicate measurements of the same concept.
Suppose, after filtering, a user only uses 4 functions; group these 4 into two thematic clusters. Display the same scale labels in full next to each statement, keep an "Not applicable / Cannot rate" option, and avoid horizontal scrolling. If item-by-item cards are used, ensure scale meaning is always visible without requiring participants to remember the previous screen. If question order is randomized, export stable item codes rather than screen positions.
This approach works better than breaking the original 12×7 matrix into 84 large buttons because it first reduces unnecessary judgments, then reworks presentation. Pew Research Center’s tips for mobile web surveys caution against large matrices and controls requiring fine motor skills, while also noting that one-question-per-page is not a fixed solution: grouping related short questions can reduce page loads. Whether to paginate should be dictated by task complexity and network conditions, not visual style.
Image, Video, and Hotspot Questions Must First Prove "Stimulus Equivalence"
Images in concept tests are not decoration; they are research stimuli. If the mobile version auto-crops the price at the bottom of the packaging, compresses fine text below readability, or shows different aspect ratios across devices, participants are not evaluating the same concept. Document the actual presentation of each image at each target size before release, checking critical information, proportion, color, loading, and alternative text.
Image taps, hotspots, or ranking require sufficiently large targets and clear selection feedback; if targets are too dense, misclicks will be interpreted as preferences. The W3C WCAG 2.2 guidance on touch target size and spacing emphasizes that smaller targets can be made less error-prone with adequate spacing. If a task cannot be performed with precision on target phones, redesign it as stepwise selection or provide equivalent controls, rather than expecting respondents to zoom.
For videos, consider muted autoplay, captions, length, load failure, and whether to allow replay. If some respondents could not watch the complete stimulus, the data should contain a recordable status, not be mixed with ratings from those who viewed it fully. Also confirm rights and any necessary privacy handling before using stimulus material.
Open-Ended Questions Are Not Forbidden—Just Ask for One Kind of Information at a Time
"Please detail what you like and dislike, and how you would improve it" on a phone demands respondents to organize three types of content at once. Replace it with a tightly decision-related prompt, e.g., "Which specific detail most influenced your choice of this packaging, and why?" Only when the follow-up would change an action should you show the next question.
Text boxes should offer contextual hints rather than character count commands, allow pasting and reasonable line breaks, and ensure the keyboard does not cover the submit button. Use input types matched to numbers, email, or dates, while still tolerating real-world formats target respondents might use. Error messages should appear near the field, explain how to fix the issue, and preserve previously entered answers elsewhere.
Pew’s analysis of open-ended question nonresponse notes that open answers inherently take more time and effort than closed ones; this does mean avoiding them altogether, but it requires them to carry unique informational value. Mobile pre-tests should read actual answers for concreteness, not simply count non-empty rates.
Page Strategy Must Balance Context, Loading, and Back Navigation
One-question-per-page promotes focus but increases page transitions and makes it harder for respondents to compare related items; a long single page facilitates review but can lead to scrolling, missing content, and burden. A common compromise: group several short questions on the same topic into one page, give high-burden tasks their own page, and keep important instructions visible alongside relevant items.
Progress bars must accommodate branching. If paths vary greatly in length, using fixed question numbers can cause sudden jumps backward or long stalls; consider coarser stage indicators based on actual path. When going back to modify answers, recompute subsequent logic and clean up outdated answers that no longer apply—do not preserve contradictory paths in final data.
If the survey is long, safe recovery matters more than smooth transitions. Test screen locking, app switching, network loss, browser page recycling, and duplicate submissions; document whether respondents can resume and avoid registering the same person after recovery as a new sample unit.
Accessibility Testing Exposes Structural Problems All Users Encounter
Questions and form controls must have programmatically associated labels; focus order should match visual order; selection, errors, and required fields should not rely on color alone; text magnified should not obscure other content; screen readers should be able to read the question, group name, options, and status. Drag-and-drop, hover, or gesture-based interactions need keyboard or simple tap alternatives.
These requirements are not just a compliance checklist. Vague labels, erratic focus, and unreachable error messages also make it easier for any average mobile user to answer incorrectly. When the target population includes people with visual, motor, cognitive, or language needs, those users should participate in pre-tests, rather than project members simulating.
Mobile Usability Testing Should Record Behavior, Not Just Satisfaction Scores
Have target participants perform tasks in their own typical environment and on their own devices, testing the longest/shortest branch, matrix adaptations, stimuli, open-ended questions, errors, and recovery. Observe where they pause, whether they see all options, whether they misclick, and how they interpret instructions; debrief afterward. Five people who all say "it was fine" won’t cover all populations, but a repeated obstacle is enough to trigger a revision.
After launching a soft launch, review data by device, OS, browser, survey version, and path:
- Successful entry, start, completion, and break-off pages;
- Median and high-percentile time per page, excluding hung sessions before interpretation;
- Item nonresponse, error retries, back-and-modify actions, and re-logins;
- Straight-lining in matrices, open-end quality, and anomalous option distributions;
- Stimulus loading, randomized groups, and completeness of critical fields.
The U.S. Census Bureau’s explanation of paradata notes that page timings, clicks, and usage logs can help identify issues, but device differences still require interpretation. Mobile completers and desktop completers are not randomly formed equivalent groups; differences in timing cannot be directly attributed to layout.
When Changing Mobile Layout, Trend Studies Need Bridging
If a historical questionnaire presented a matrix on desktop and the new mobile version splits it into item-by-item cards, identical wording does not guarantee measurement equivalence. GESIS’s questionnaire mode transition guide cautions that different devices and presentations can affect comparability and recommends documenting device, login, and presentation format.
For key trend indicators, consider randomly assigning old vs. new presentation during a transition period, or keeping one panel on the old version as a bridge. Compare distributions, missingness, duration, and subgroup differences before deciding whether trends can be linked. If randomization is not possible, at least record device and version and limit interpretations. Do not silently change scale points, option order, or question meaning in the name of "mobile optimization" and then write a break as an attitude shift.
Implementation Focus in Wenjuanpai: Reducing Cognitive Load and Evidence from Real Devices
When using Wenjuanpai to design web surveys, first use screening, skip logic, and answer piping to reduce the number of questions each respondent truly needs to evaluate; then preview and submit test responses on target phones and browsers. For matrices, images, hotspots, ranking, and long text, validate each against the task-budget approach described above, and keep test data separate from formal data collection.
After release, if the data retain device, path, and version information, use them for stratified monitoring—but never treat device labels as causal conclusions. Connect survey version, page, device, final state, and quality flags to soft launch acceptance to determine whether problems originate from recruiting, population differences, item wording, or interaction.
When mobile design is done well, it doesn’t mean that mobile completion rates are necessarily higher than desktop—it means the device no longer inadvertently alters item meaning, answer options, or data structure. Final acceptance should answer: Did participants on the smallest screens see the same decision-relevant information as desktop users? Can every valid answer be confirmed not to be induced by layout? Did those who broke off leave a diagnosable state? Only if these hold does "mobile-friendly" become part of research quality, not just page adaptation.
Evidence Basis for Mobile Survey Design and Mode Effects
- U.S. Census Bureau: Proposed Standards and Guidelines for Mobile Survey Instrument Design
- Pew Research Center: Tips for Creating Web Surveys for Mobile
- Pew Research Center: Open-ended Questions and Item Nonresponse
- GESIS: Questionnaire Design for Mode Transition and New Devices
- U.S. Census Bureau: About Paradata
- W3C WAI: Target Size (Minimum)