Education research must first answer "what needs to change," and only then "is the course effective?" Satisfaction, perceived gains, assignment completion, and ability change operate at different levels of evidence: satisfaction shows whether the experience met expectations, completion shows whether learners finished the process, and pre/post-tests show whether outcomes changed for the same people. Only credible concurrent comparisons or randomized controls come closer to answering whether the change was caused by the course. Combining all four into a single "course effectiveness score" allows good experiences to masquerade as learning gains and undervalues truly effective but challenging courses.
Therefore, an executable course evaluation should not begin with "how many questions do we need to ask" but should first clarify the decision: whether to revise the instruction in Unit 2, adjust teacher support, decide whether to scale up the pilot, or decide whether to renew the purchase. Different decisions require different evidence. If the goal is only to fix the learning path, process data, short surveys, and observations are already valuable; if you want to publicly claim that the course raised test scores, you need more rigorous outcome measures, baselines, comparison groups, attrition handling, and inferential boundaries.
First, Break Learning Evidence into Four Layers
- Conditions layer: Are devices, internet, time, prerequisite knowledge, language, and accessibility support in place? These factors determine whether learners have a fair opportunity to participate.
- Implementation layer: Was course content delivered as planned, did teachers complete key activities, and how much instruction, practice, and feedback did learners actually receive? Without implementation records, low outcomes may stem from course design or from the course not being fully executed at all.
- Experience and process layer: Were objectives clear, was difficulty appropriate, was feedback timely, where did learners get stuck, and did they feel comfortable seeking help? This layer is best suited for surveys, interviews, and task observation.
- Outcomes layer: Did knowledge, skills, transfer tasks, or subsequent behaviors aligned with objectives change? Outcomes must be measured with verifiable instruments and kept distinct from longer-term claims about grades, progression, or employment.
The four layers are related but not interchangeable. Learners feeling that "the explanation was clear" does not guarantee they can independently solve new problems; rising test scores do not show that the course was engaging enough to sustain persistence. Managers should set separate indicators—for example, implementation rate, key task completion rate, difficulty points, target skill performance, and dropout rate—rather than pursuing a single composite average.
Who Is Qualified to Answer What: Don't Let One Role Testify for Everyone
Learners are best qualified to report on understanding, effort, emotions, barriers to seeking help, and specific experiences; teachers can report on preparation burden, classroom execution, common errors, and resource fit; parents can describe family support, communication, scheduling, and observable changes at home; administrators or purchasers understand deployment, training, cost, and organizational goals. Teachers cannot accurately answer for each student's anxiety, and parents cannot assert ability gains solely from a child's "willingness to attend."
An actual survey can first confirm role, course, class, and participation stage before branching into relevant modules, keeping only a few common questions. For "whether teacher feedback was timely," ask only those who submitted work requiring feedback; for parent communication, cover only those who actually received notifications. The denominator must follow the experience, rather than counting "not applicable" as dissatisfaction.
Work Backward from an Observable Learning Objective to Measurement
"Improve math ability" is too broad. It can be rewritten as: "After completing the four-week course, can learners select an appropriate proportional relationship for a new context and explain their reasoning without step-by-step prompts?" Then design a task aligned with this objective and administer versions of comparable difficulty and scoring rules before and after the course. If the pre-test and post-test items are identical, score gains may reflect memorization; if the two versions differ in difficulty, test differences may be mistaken for learning changes.
Experience questions should also target actions. For example, rather than simply asking "was the course satisfactory," ask learners where they first felt uncertain, how they tried to recover, whether they found feedback, whether that feedback explained their error, and whether they could complete the next problem independently. Use open-ended probes to uncover mechanisms and structured indicators to estimate prevalence; first confirm through cognitive pretesting how learners of the target age understand the items, then administer formally.
Case of 200 Learners: A 16-Point Gain Among Completers Cannot Represent the Whole Class
Below is an illustrative case. A four-week online course had 200 enrollees who completed the pre-test, with a total pre-test average of 52. At the end, only 160 completed the post-test, while 40 were lost to follow-up. Among post-test completers, the average pre-test score was 55 and the average post-test score was 71, so the average gain for these completers was 16 points.
If the report simply says "learning scores increased by an average of 16 points," it obscures two issues. First, the attrition rate is 20 percent; second, the 40 lost participants had an average pre-test score of only 40, clearly lower than the completers' 55. In other words, those who remained started higher, and completers are no longer an equivalent subset of the original 200. Attrition not only reduces sample size but also changes sample composition.
The team can conduct a transparent sensitivity analysis, but must not package assumptions as observed facts. If only as a scenario assumption, setting the final score of the 40 lost participants at their pre-test score of 40—that is, assuming no improvement—would yield a cohort final average of:
(160 × 71 + 40 × 40) ÷ 200 = 64.8
Compared with the cohort pre-test average of 52, the scenario gain is 12.8 points. Under a more unfavorable stress scenario, assuming the lost participants' final average was 5 points below their pre-test score, i.e., 35, the cohort final average would be:
(160 × 71 + 40 × 35) ÷ 200 = 63.8
Corresponding to a gain of 11.8 points. Both results are lower than the completers' 16 points. These are not the true answers for the missing values, nor are they recommendations to replace full analysis with single imputation; their value lies in letting decision-makers see how sensitive the conclusion is to unobserved outcomes. Formal effectiveness studies should also state the missingness mechanism, compare known baselines between responders and non-responders, and adopt missing-data handling methods appropriate to the design and data.
Even with Rising Scores, You Cannot Automatically Attribute the Change to the Course
The 16, 12.8, or 11.8 points above are merely descriptions of change. Without a concurrent comparison group, practice effects, natural maturation, concurrent school instruction, test preparation, selective persistence, and changes in testing environments can all explain part of the gain. The conclusion can be written as "completers improved by an average of 16 points on this measure; under two attrition scenarios, the full cohort change was 11.8–12.8 points," but not as "the course raised scores by 16 points."
If the decision concerns scaling up investment, randomly assign course and comparison versions among eligible participants, or randomly sequence implementation at the class level. Compare concurrent outcomes between the two groups and respect the original assignment in the analysis. When randomization is impossible, seek a plausible comparison group, document key baselines, and adjust carefully, but unmeasured differences may still remain. The U.S. Institute of Education Sciences' What Works Clearinghouse lists attrition, baseline equivalence, and confounding as key criteria for study quality, showing that "having a pre- and post-test" alone is insufficient to establish a causal claim.
Classes, Teachers, and Schools Make Observations Correlated
Students of the same teacher share explanation styles, assignment pacing, and classroom climate; schools share admissions, resources, and evaluation systems. Collecting 500 students does not yield 500 fully independent observations. If the new course version is taught only by Teacher A and the old version only by Teacher B, then course version and teacher are completely confounded, and any difference cannot be attributed to either factor.
Designs should arrange for multiple teachers or classes to appear in each condition where possible, and preserve identifiers for school, class, teacher, and course version; analyses and uncertainty estimates should account for the clustered structure. Reports should also present implementation differences side by side: if average outcomes did not improve but classes that faithfully completed core activities showed larger gains, that is only an implementation cue worth verifying, not a license to cherry-pick high-performing classes as proof of effectiveness post hoc.
Attrition Analysis Should Track "Who Did Not Come Back" and "Where They Left"
End-of-course surveys naturally reach only those who remain. Genuinely struggling learners may have dropped out by Week 2, so satisfaction can be systematically inflated. Projects should record, from roster creation onward, the status of invitations, consent, pre-test, actual participation, each key milestone, post-test, and analysis inclusion, assigning mutually exclusive disposition codes to each type of exit. The attrition rate should at minimum use those initially eligible for analysis as the denominator, while also comparing overall and differential attrition between groups.
Reducing attrition is more important than repairing it afterward: explain future contact at the start; schedule measurement at times not affected by grading; use age-appropriate, short, and clear questionnaires; offer reasonable makeup sessions for absentees; retain the minimum contact details permitted by compliance; and allow those who stop the course to voluntarily provide exit experiences. Do not force completion to create a veneer of completeness, and do not delete pre-test and participation records simply because the post-test is missing.
When Involving Minors, Ethics Is Not Just a Consent Form
UNICEF's guidance on research involving children emphasizes children's dignity, rights, and well-being, treating harm and benefit, informed consent, privacy, and confidentiality as ongoing judgments rather than one-time procedures. Specific projects should undergo professional ethics and compliance review according to local regulations, institutional requirements, and research risk. Where guardian permission is required, that does not mean children's own age-appropriate information and assent can be ignored; a child's refusal or withdrawal must not affect grades, teacher attitudes, or access to resources.
Power relations in schools are especially important. If the classroom teacher stands nearby collecting evaluations of their own course, students will find it hard to believe negative answers have no consequences. A non-teaching staff member can explain the purpose, separate identity from evaluation data, allow skipping sensitive questions, and specify who can see raw responses. Do not collect names, exact addresses, full grades, or sensitive family information merely because they "might be useful later." Even with names removed, cross-tabulations in small classes can re-identify individuals through grade, class, and specific experiences; combine small cells or cease publication.
How to Write an Education Research Report That Supports Decisions
The report's first page should state the decision conclusion, then elaborate by evidence level. The following sentence templates can be used:
- Observed fact: The planned completion rate for Unit 2 was 62 percent, the lowest among the four units; dropouts were more concentrated in the low-scoring prerequisite test group.
- Participant explanation: "Examples skip straight to the formula" and "I didn't know how to ask for private help" recurred in interviews, suggesting a need to examine the explanation span and help-seeking channels.
- Actionable step: Add one diagnostic exercise, one worked error example, and a private help request entry point to Unit 2, keeping everything else unchanged.
- Verification plan: In the next round, predefine key task completion as the primary process indicator and an equivalent post-test as the outcome indicator, while monitoring dropout rates and help-seeking burden.
- Inferential boundary: This round has no concurrent comparison, so score changes cannot be attributed to the course; end-of-course experiences represent only those who actually responded.
Every figure should state the population, denominator, time point, missingness, and course version. Publicly ranking raw scores of learners, classes, or schools can harm participants and readily misread background and implementation differences as ability differences. For small-sample groups, prefer reporting ranges and uncertainty over using decimal points to create a sense of precision.
Pre-Publication Checklist
- Research decisions, learning objectives, primary outcomes, and success thresholds are specified before data collection.
- Condition, implementation, experience, and outcome indicators are separated; satisfaction does not replace learning effectiveness.
- Learners, teachers, parents, and administrators answer only questions about what they can observe.
- Pre-tests, post-tests, or equivalent tasks align with learning objectives, and scoring rules are verifiable.
- The roster can track invitation, participation, dropout, attrition, and final analysis status.
- The report fully documents attrition rates and compares known baselines between responders and non-responders.
- Causal claims have a credible comparison; without one, only changes and mechanistic cues are described.
- Teacher, class, and school clustering enters the design, analysis, and conclusion boundaries.
- For research involving minors, age-appropriate information, assent or consent, withdrawal, protection, and data minimization are completed.
- The report publicly presents observed facts, participant explanations, actions, verification plans, and limitations.
Before designing, you can use the research question framework to turn "evaluate the course" into a verifiable decision; when planning baselines, indicators, and grouping, combine it with the pre-collection analysis plan; when dealing with those who did not complete the post-test, continue with questionnaire missing data analysis and nonresponse bias assessment. The most valuable output of education research is not proving that a project is "generally good" but showing which link in the evidence chain is already established, which remains uncertain, and how to verify it with the smallest change in the next round.
Methodological Basis and Further Reading
- IES What Works Clearinghouse: Procedures and Standards Handbook, Version 5.0: Randomized designs, attrition, baseline equivalence, missing data, and reporting standards in education research.
- IES What Works Clearinghouse: Standards Briefs: Concise explanations of attrition, baseline equivalence, and confounding.
- UNICEF Innocenti: Ethical Research Involving Children: Dignity, rights, well-being, harm and benefit, consent, privacy, and confidentiality in research with children.
- UNICEF: Policy on Ethics in Evidence Activities Involving People: Ethics review, protection, risk management, consent, and data management requirements across the full lifecycle of evidence activities.