E-commerce experience research should not only ask for satisfaction after delivery, as this systematically misses users who abandon searches, find product information insufficient, abort checkout, encounter payment failures, or actively cancel. A more robust design uses a specific shopping task or order as the unit and establishes distinct sampling frames at each stage: behavioral data shows where users drop off, while surveys and interviews explain what they intended to accomplish, how they interpreted the page, and why they continued or exited. The final output should be attributed to a specific, actionable step in the journey, rather than a generic “platform experience score.”
Task success, efficiency, and satisfaction represent three different types of evidence. High page conversion does not necessarily mean consumers clearly understood prices and terms, and high satisfaction does not guarantee that orders were processed successfully. NIST’s usability testing framework considers task completion and errors, completion time, and subjective satisfaction simultaneously; e-commerce research should similarly align these dimensions with actual business outcomes.
First Define the Unit of Analysis: Session, Task, Order, or Customer
A single customer might search on mobile, order on desktop, have the order split into two packages, and then initiate a partial return. Sessions are suited for identifying page-level drop-offs, shopping tasks for understanding cross-device comparisons, orders for linking payment and fulfillment, and customers for analyzing long-term relationships. These four units cannot be simply merged via a single user ID.
For conversion research, define a task as “finding an item matching specific criteria within a given timeframe” and record entry point, search terms, filters, comparisons, product detail views, add-to-cart actions, and exits. For fulfillment research, use the order or package as the unit, documenting promised time, actual delivery, damage or loss, and signature confirmation. After-sales research should be tied to return/exchange or complaint tickets, distinguishing between submission, review, return shipping, refund, and final resolution. Relational metrics can be measured after events but cannot substitute for event-level evidence.
Including authorized category, timing, and order status details in the invitation can reduce recall bias. Research on questionnaire construction by the U.S. Census Bureau notes that ordinary consumer purchases are increasingly omitted as the recall interval lengthens, and their occurrence tends to be remembered as more recent. Instead of asking “How is shipping usually?” it is better to ask about “the order you just received” or “the checkout you aborted last night”; also, display only the minimum order information required to complete the task.
Five Distinct User Groups Require Different Entry Points and Questions
- Searched/Browsed but did not add to cart: Determine if they found a suitable item, if information was sufficient, or if they were only comparing for future use; not all exits are failures.
- Added to cart but did not checkout: Distinguish between saving for later, order consolidation, price changes, stock issues, fees, login problems, and delivery conditions.
- Attempted checkout but did not pay: Record the last successful step, error messages, payment methods, discount understanding, and whether they switched to another channel.
- Paid and order was fulfilled: Evaluate clarity of promises, usefulness of tracking, product conformance to expectations, and how any issues were handled.
- Processing a return/exchange or complaint: Break down the process into application requirements, evidence submission, processing time, refund receipt, and whether the issue was truly resolved.
Each group should only answer modules relevant to their actual experience. Users who never contacted customer service cannot rate the resolution capability of customer service, and users who never tried a coupon should not be in the denominator for “coupon usage success rate.” Exit interviews should be brief and easy to dismiss; they should not become a barrier themselves through consecutive pop-ups. For logged-in users, set a maximum contact frequency to avoid surveying them repeatedly at payment, shipping, delivery, and after-sales stages for the same order.
400 Checkout Abandoners Saying “Price Too High” Does Not Necessarily Mean You Should Lower Prices
The following is an illustrative diagnosis; numbers do not represent a real platform. A team sampled invitations from qualifying checkout abandonment sessions and received 400 valid responses. 160 people first selected “final price higher than expected.” When asked where their last price expectation was broken, 70 said essential shipping or service fees only appeared later in the process, 52 said coupon conditions differed from their earlier understanding, 8 encountered surcharges for their chosen payment method, and 30 found that the price of a substitute item was different after specification or stock changes. The remaining 240 people: 82 encountered payment failures, 64 had no available address or delivery time, 52 were simply deferring their decision, and 42 cited other reasons. Thus, “price high” is only a top-level label: 70 and 8 require fee transparency, 52 require clarity on coupon eligibility and failure reasons, and 30 need transparent specifications and substitution logic. Genuine price sensitivity can only be studied under conditions of identical items, total price, and delivery terms.
These proportions only describe the feedback sample. Those willing to answer the exit survey may be more dissatisfied than the entire population of abandoners, and those responding to the invitation may be concentrated among logged-in users. Reports should present the number of qualifying sessions identified, invitations displayed, surveys started, valid responses, and mobile/guest/channel coverage; they should not simply divide 160 by 400 and state that “40% of users churned due to price.”
Translate Subjective Reasons into Verifiable Hypotheses
An open-ended response like “the promotion is complex” can be broken down into several distinct issues: invisible eligibility criteria, non-stackable offers, incorrect threshold calculations, entry points that are hard to find, vouchers expiring before use, and inexplicable system failure messages. Each mechanism has different verification evidence:
- Invisible eligibility: Test whether participants can state applicable items and thresholds before making a choice;
- Hard-to-find entry: Observe whether representative users can complete the coupon-claiming and redemption task without a prompt;
- System calculation error: Check the rule engine, order logs, and customer support tickets;
- Rules too complex: Compare the impact of a simplified design on correct understanding, total price expectation, and task completion;
- True price sensitivity: Run pricing or promotion experiments under compliant and consistent conditions.
Behavioral logs tell the team which step failed, but not necessarily how the consumer interpreted it; self-reported reasons explain the “why” but are subject to memory and post-hoc rationalization. If these two sources conflict, examine event matching, question wording, and sample selection. For instance, if logs show that a coupon was applied but the consumer says “there was no discount,” the issue may be unclear total price feedback, not a calculation failure.
A 12,000-Session Experiment: Transparent Fees Can Also Increase Conversion
Based on the above diagnosis, the team proposed a solution: display the estimated total item amount, essential fees, available promotional offers, and any pending delivery conditions before the user enters the checkout flow. The following example continues with sample data. 12,000 qualifying sessions were randomly assigned to the old version and the new version, 6,000 each. The old version saw 4,200 completed payments, a 70% conversion rate; the new version saw 4,380, a 73% conversion rate.
The difference is three percentage points. Using a simple approximation for the difference between two independent proportions, the 95% confidence interval is roughly 1.4 to 4.6 percentage points. With proper randomization and sample size, this provides stronger causal evidence than “satisfaction improved after the redesign,” but one must still check for errors in split assignment, any price or stock anomalies during the experiment, and adherence to pre-specified exclusion rules in the analysis.
Conversion cannot define success alone. The new version should also be monitored for rapid cancellations after payment, refunds, fee-related support contacts, accidental purchases, average order value, and repeat purchases. If higher payment conversion was achieved by pre-selecting add-ons, hiding terms, or making cancellation more difficult, then the consumer task has not improved. The U.S. Federal Trade Commission’s report on dark patterns specifically discusses practices such as delayed disclosure of essential fees, hiding key terms, presenting pre-selected options, and creating cancellation obstacles; the research goal should be to enable consumers to complete choices that align with their intentions while understanding the conditions.
Fulfillment Surveys Must Separate Promise, Process, and Outcome
“Shipping satisfaction” conflates at least three distinct aspects: whether the promise at order time was clear, whether information during transit was useful, and whether the final package arrived on time and in good condition. Actual delivery in three days is a failure if the promise was two; actual delivery in five days might meet expectations if the promise at purchase clearly indicated seven. When comparing carriers and warehouses, stratify by service level, region, category, holidays, and exceptional weather events.
For multi-package orders, ask participants to specify which package had the issue. Missing items, damage, incorrect product, delay, and delivery failure have different chains of responsibility. For problematic orders, connect objective timestamps, tracking data, and support tickets with questions about whether information was clear, if repeated contact was necessary, and whether the issue was ultimately resolved. However, a closed support ticket does not automatically mean the consumer considers the issue resolved.
The Endpoint for After-Sales Research is Problem Resolution, Not Process Closure
For the return/exchange path, separately record whether the application was successful, what evidence was required, the time to first decision, return shipping, refund promise, actual arrival, and whether trust was ultimately restored. Users still in the process cannot be forced to rate the final resolution; conversely, only surveying after a successful refund will miss those stuck in the process. Trigger short surveys based on status and conduct a follow-up sample after the case is closed.
Low-frequency but severe issues cannot be prioritized solely by incidence rate. Duplicate charges, inability to cancel, privacy breaches, incorrect items, or high-value refunds may affect few users but cause significant harm. Prioritization should combine the number of affected users, the degree of task blocking, economic and trust risk, controllability of responsibility, and remediation cost. Start with diagnostic research on high-relevance steps, but any formal rollout still requires an experiment or a comparable pre/post test.
Boundaries When Comparing Across Channels and Time Periods
First-party apps, web, mini-programs, third-party storefronts, and in-store pickup all have different functionalities and user bases; they cannot be combined into one “e-commerce experience” and directly ranked. Version release timing, promotion periods, out-of-stock events, peak shipping capacity, pricing strategies, and invitation placement also alter the sample. Methodology reports must retain details on channel, device, user state, category, event time, survey delay, invitation probability, and unweighted base counts.
Quarterly relationship surveys are suitable for tracking trust, value, and alternative platforms, while post-transaction surveys are suitable for identifying order-specific issues. Relationship metrics may be influenced by the most recent anomalous experience, whereas transactional feedback only represents the invited event. The two should be used to interpret each other, not converted from one to the other. Public ratings come from proactive, vocal users; they are useful for spotting new themes and sentiment shifts but should not directly replace structured surveys with a defined sampling frame.
A Directly Actionable E-commerce Research Checklist
- Clearly define the analysis unit, target task, category, channel, and event time window.
- Establish independent sampling frames for browsing exits, cart abandonment, checkout, payment, fulfillment, and after-sales stages.
- For each module, first ask if the step was needed, attempted, and completed; then ask about difficulty and reasons.
- Minimize order fields when linking to behavior; clarify whether anonymous feedback will be linked back to an individual account.
- Record the number of invitations seen, surveys started, completed, quality-passed, and unique users.
- Decompose broad reasons into mechanisms that can be falsified through logs, task tests, or experiments.
- When running experiments, monitor conversion alongside accidental purchases, cancellations, refunds, complaints, and subsequent behavior.
- Report results by channel, category, and fulfillment conditions; avoid automatic ranking based on small base sizes.
- After a redesign, maintain the same invitation criteria and document promotion, version, stock, and shipping capacity events.
To reconstruct the full shopping journey, refer to the guide on customer journey research surveys; for validating search, promotion, and checkout tasks, see the questionnaire design after usability testing; for analyzing large volumes of exit reasons and customer service text, follow the open-ended response coding methodology to preserve original quotes, codes, and counterexamples. A publishable e-commerce conclusion should include the specific task, journey stage, actual denominator, behavioral evidence, user explanation, and validation method.
Methodological Sources and Further Reading
- NIST: Usability Testing: Evidence of completion, errors, efficiency, and satisfaction in representative tasks.
- U.S. FTC: Bringing Dark Patterns to Light: Delayed fees, hidden terms, pre-selected options, and cancellation obstacles.
- U.S. Census Bureau: Survey Questionnaire Construction: Recall of consumption events and reference periods.
- AAPOR Transparency Initiative: Disclosure of sample source, invitation, administration, disposition, and unweighted case counts.