Of 1,000 completed responses, 96 were unusually fast, 118 showed low variance in long matrices, and 82 failed attention checks. If any single signal were treated as a deletion criterion, many genuine respondents with atypical response styles would be wrongly removed; if only attention checks were used, fraudsters who carefully cater to screening questions could easily pass. The target of quality detection is not responses that “look unlike the average person,” but records that fail to provide credible measurement for the research question.
A practical approach is to first distinguish confirmed invalid responses, high-risk responses requiring multi-signal review, and borderline records that are retained but used for sensitivity analysis. Rules are defined before data collection, thresholds are calibrated against real paths and devices, manual review is blinded to primary outcome direction, and raw data, signals, processing reasons, and cleaned versions are all preserved.
“Low quality” is not one cause; at minimum, split into five categories
| Type | Data manifestation | Appropriate handling direction |
|---|---|---|
| Ineligible | Actually not part of the target population or role | Return to screening and recruitment; don’t label as poor attention |
| Duplicate or fraudulent | Same participant repeatedly compensated, fabricated identity or experience | Verify identity and duplication chain; retain evidence of action |
| Technical invalid | Stimuli not displayed, logic skips wrong, upload or translation failures | Fix the survey and assess affected items; don’t blame the respondent |
| Careless or irrelevant | Did not follow instructions, mechanical answers, meaningless text | Combine content, process, and consistency evidence |
| Rare but credible | Extremely fast, straight-lining, extreme behavior, or unconventional networks | Corroborate experience; don’t auto-delete simply because uncommon |
Curran’s methodological review of careless invalid self-report data further distinguishes language or comprehension deficits, deliberate misreporting, and careless insufficient effort. These three can produce similar surface signals yet require different remedies: translation issues call for item revision, incentive-driven qualification gaming calls for revised recruitment and verification, and fatigue in the latter half calls for a lighter survey burden.
First build a signal dictionary, then decide hard failures versus soft reviews
| Signal | What it may indicate | Why it cannot stand alone as evidence |
|---|---|---|
| Short duration | Skimming, automation, or low effort | Familiar respondents can be fast; paths, devices, and languages differ |
| Same column in matrix or low variance | Mechanical responding | Can be genuine agreement when items are unidirectional or constructs are similar |
| Contradictory answers | Comprehension failure, misreporting, or inattention | Time frames, conditions, or item wording may actually differ |
| Irrelevant open text | Copying, generation, or low effort | Short answers may be complete; language ability cannot be equated with quality |
| Attention check failure | Failure to follow one clear instruction | Misclicks, accessibility issues, translation problems, or local lapses can happen |
| Matching device/IP/fingerprint | Clue of duplicate participation | Corporate, school, household, and proxy networks can share egress points |
| Rating despite failed stimulus exposure | Responding without exposure or anomalous logs | Client-side caching or tracking failures must be ruled out first |
Two studies by Meade and Craig compared attention checks, consistency of regular items, multivariate outliers, response times, and self-reported effort, finding that different indicators detect different patterns of random and non-random carelessness, with effectiveness dependent on data properties. There is no “universal quality score” that works alone across all surveys.
Case of 1,000 responses: 210 flagged, only 65 ultimately excluded
The following illustrative data are fictional and do not represent Wenjuanpai, any vendor, or industry benchmarks. Rules and thresholds were set via a soft launch before formal collection, and the primary business outcome was hidden from the review interface.
The system first confirmed 28 hard invalids: 12 were duplicate submissions from the same controlled invitation and participant account, with the earliest valid record retained per preset rules; 16 had confirmed, via combined server and client logs, that core concept stimuli were not successfully presented yet all stimulus ratings were completed. No conclusion was drawn solely from matching IPs or a single loading event.
Of the remaining 972, 210 triggered at least one soft signal. Signal counts overlap: 96 unusually fast, 118 low variance in long matrices, 74 contradictions on key experiences, 53 clearly irrelevant open text, and 82 attention check failures. 142 triggered exactly one signal; after review, all were retained with flags; 68 triggered at least two distinct dimensions and underwent dual review.
| Review tier | Count | Handling |
|---|---|---|
| Verified hard invalid | 28 | Excluded; raw records and evidence retained |
| Exactly one soft signal | 142 | Retained; no single-threshold auto-deletion |
| Multiple signals with evidence supporting invalidity | 37 | Excluded after dual review |
| Multiple signals but experience explainable | 23 | Retained |
| Multiple signals with unclear technical boundary | 8 | Retained in primary analysis; excluded in sensitivity version |
The primary analysis ultimately retains 1000 − 28 − 37 = 935 responses. If “any soft signal triggers deletion,” soft signals alone would remove 210 responses, most without multi-evidence support; if only the 82 attention check failures were excluded, duplicates and sycophantic responders who passed attention checks would be missed.
Cleaning not only changes sample size but can also shift conclusion direction
Among the original 1,000, 540 supported the new plan, at 54.0%. Of the 65 excluded, 20 supported it, so the primary analysis yields (540 − 20) ÷ 935 = 55.6%. Of the 8 technical-boundary records, 4 supported it, yielding a sensitivity version of 516 ÷ 927 = 55.7% after further exclusion. In this example, cleaning raises the proportion by about 1.6 percentage points, and boundary handling barely changes the conclusion.
This does not mean cleaning always improves outcomes. Low-quality responses may mechanically agree or randomly dilute differences; among rare populations, a small number of faked qualifications can dramatically alter proportions. Reports should present the original, pre-specified primary analysis, and key alternative rules, rather than only the version that best fits expectations.
Duration thresholds must be based on actually observed visible paths
Half the median total time is not a universally reliable cutoff. Respondents who saw only 12 items cannot share a threshold with those who triggered 60; desktop keyboard input differs from mobile open text. A better process:
- Use pre-tests with target users to estimate reading, judgment, and input burden for each major path;
- Remove clearly backgrounded time from page-level logs, but retain raw values;
- Examine distributions separately for path, device, language, and key assistive technology users;
- Treat extremely short duration as a risk signal, then combine with content and consistency;
- Monitor distribution drift during formal collection; any threshold change must leave a version and rationale.
An expert may quickly recognize familiar options and give an accurate fault description; another person may show normal total time but leave the page backgrounded while copying irrelevant text. Total minutes cannot distinguish the two.
Before flagging straight-lining, ask whether the scale allowed real variation
Selecting “agree” for all five unidirectional items describing reliability may indicate stable attitudes; lack of variation across a 30-row matrix with mixed directions and multiple distinct experiences is more worthy of scrutiny. Combine longest run of identical values, within-person variance, consistency on reversed or semantically similar items, response time, and use of “not applicable,” while recognizing that difficult reversed items can fabricate contradictions.
Do not add more repeated or reversed items to catch straight-lining. Added burden creates new carelessness. Prioritize shortening matrices, making each row measure only one concept, providing legitimate “not applicable,” and verifying in cognitive pretests that respondents interpret the scale as designed.
A single attention check both under-detects and over-flags
In Pew Research Center’s study of multi-source online surveys, 84% of suspicious responses defined by other serious behaviors passed a standard instructional attention check; abnormal speed similarly failed to catch most problems, and IP checks alone were insufficient to identify duplicates. This shows that incentive-driven participants may be highly attentive to screening and traps while simply not providing genuine experience.
Attention checks can be a local signal but should not shame respondents or deliberately use convoluted wording to set traps. Verifiable experiences relevant to the research topic, actual stimulus exposure, reasonable logical consistency, and open-ended reasons typically provide richer evidence. A single failure may warrant a reminder or review; exclusion depends on pre-specified severity and other signals.
When AI reviews open text, guard against treating “doesn’t look like a standard answer” as low quality
“Didn’t work,” “too expensive,” “stuck at payment” may be very short yet fully answer the reason; non-native speech, colloquialism, dialect, typos, or assistive input cannot be automatically equated with gibberish. AI can help identify large-scale repetition, off-topic templates, and similar fragments across responses, but the original prompt, context, and language must go to human review.
Reviewers should not see respondents’ direction of support for the primary proposal, nor should negative, strongly worded, or minority viewpoints raise risk scores. The quality standard for open text is relevance to the question, readability, and compatibility with the response path—not politeness of tone or length.
Duplicate detection must balance privacy and shared environments
The CHERRIES checklist for internet e-surveys recommends specifying whether cookies, IPs, logs, or login were used to prevent duplicates, the restriction period, and which record is kept when multiple exist. It also advises disclosing duration thresholds and incomplete-response handling. Current practice should follow data minimization: collect only signals needed for research and fraud prevention, set retention periods and access controls, and recognize that hashing or fingerprinting does not lose identification risk just because it is unreadable.
The same IP may be a corporate, campus, household, or mobile carrier egress; different IPs may belong to the same person switching networks. Reliable judgment typically requires a compliant combination of invitation tokens, accounts, time, devices, response similarity, payment, or vendor information, with a path for dispute review. Using privacy-protection tools per se cannot be defined as cheating.
Upstream prevention works better than ex-post blacklist expansion
- Do not publicize all qualifying answers in invitations; use neutral, cross-verifiable recent behaviors in screening;
- Reduce unnecessary length and matrix items to avoid fatigue in the latter half for genuine respondents;
- Ensure stimuli, branching, translations, mobile, and accessibility paths pass testing first;
- Align incentives with the real burden, while preventing the same person from collecting repeatedly;
- Monitor signals by source, quota, device, and collection time without permanently labeling channels;
- Validate rules on known test cases and human-reviewed samples, estimating false positives and false negatives.
The ESOMAR/GRBN online sample quality guideline places respondent verification, fraud prevention, response effort, category exclusions, sample selection, blending sources, weighting, routing, profiling, and screening in one system. Quality is not running a script after the survey is closed; it is a full-process matter from sample source to analytical version.
How to write a reviewable cleaning report
Wenjuanpai can record page-level duration, logic paths, stimulus exposure, and open answers, and bring multiple quality signals into analysis. Rules govern flags and explicit technical actions; boundary cases undergo human review under the same standard; raw, primary, and sensitivity data sets are versioned and do not overwrite original responses.
This case can be reported as: “Of 1,000 completed responses, 28 were excluded under pre-specified hard rules due to verified duplicates or core stimuli not being shown. Of the remaining 972, 210 triggered at least one soft signal; 142 with a single signal were retained, and 68 with multiple signals underwent dual review blinded to the primary outcome, resulting in 37 exclusions and 31 retentions. The primary analysis includes 935 responses, support at 55.6%, up 1.6 points from the original 54.0%; excluding 8 technical-boundary records yields 55.7%, showing stable conclusions.”
Such a report allows readers to see rules, overlaps, human judgment, and result sensitivity. Writing only, “6.5% low-quality samples were deleted after cleaning,” is neither reviewable nor does it distinguish technical failures, genuine minorities, and fraud under one label.
Readers may also consult soft-launch monitoring to detect signals earlier, use a survey data-cleaning workflow to manage versions, and reference survey length and cost to reduce fatigue sources.