Watson-Glaser Critical Thinking Appraisal Interpreting Evidence and Data Questions and Answers 5 — Questions and Answers
Question 1: A clinical trial reports that a new drug reduced symptoms in 40% of patients compared to 30% on placebo. The p-value is 0.04. What is the most accurate interpretation?
- The drug is definitely effective and should be adopted immediately
- The result is statistically significant, suggesting the difference is unlikely due to chance, though clinical importance should also be assessed (Correct answer)
- A p-value of 0.04 means there is a 4% chance the drug works
- The placebo had no effect at all
Correct answer: The result is statistically significant, suggesting the difference is unlikely due to chance, though clinical importance should also be assessed
Statistical significance at p<0.05 indicates the result is unlikely to be random chance, but clinical significance (the size and practical value of the effect) must also be evaluated.
Question 2: A graph shows violent crime falling every year for a decade. A politician says 'our new policing program, launched 2 years ago, is working.' What is the key interpretive issue?
- Crime statistics are inherently unreliable
- Crime was already declining before the program started, so attributing the trend to the program is premature (Correct answer)
- The program should be expanded based on this evidence
- Violent crime always falls over time
Correct answer: Crime was already declining before the program started, so attributing the trend to the program is premature
The pre-existing downward trend existed before the intervention, making it impossible to credit the program without isolating its specific effect.
Question 3: A study finds that people who floss daily have fewer heart attacks. Before concluding flossing protects the heart, what must be considered?
- Flossing technique may vary by individual
- People who floss daily may share other health habits that reduce heart attack risk (Correct answer)
- Dentists may be biased in reporting heart data
- Heart attack victims cannot floss
Correct answer: People who floss daily may share other health habits that reduce heart attack risk
Daily flossing may be a marker of overall health-consciousness, meaning better diet, exercise, and other habits — not flossing itself — could explain the outcome.
Question 4: A report states that School X's test scores improved 20 points after adopting a new curriculum. However, 30% of the lowest-scoring students transferred out that year. What does this suggest?
- The curriculum is highly effective
- The score increase may be partly or entirely explained by a change in the student population, not the curriculum (Correct answer)
- Transferring students always improves scores
- The curriculum should be adopted nationwide
Correct answer: The score increase may be partly or entirely explained by a change in the student population, not the curriculum
When lower-performing students leave a group, average scores rise mechanically — this is sometimes called compositional change or a selection effect.
Question 5: A researcher publishes data showing a positive result. Ten other teams tested the same hypothesis and found no effect, but did not publish. What problem does this illustrate?
- The published researcher is dishonest
- Publication bias, where positive results are more likely to be published, making the evidence base misleadingly favorable (Correct answer)
- The other teams used inferior methods
- Statistical significance guarantees a real effect
Correct answer: Publication bias, where positive results are more likely to be published, making the evidence base misleadingly favorable
Publication bias skews the literature toward positive findings, making interventions appear more effective than the full body of evidence would support.
Question 6: A data table shows that 9 out of 10 dentists recommend Brand X toothpaste. What question is most important to ask before accepting this as strong evidence?
- What country the dentists are from
- How the dentists were selected, how many were surveyed, and what question they were actually asked (Correct answer)
- Whether Brand X is the most expensive option
- Whether dentists brush their own teeth twice daily
Correct answer: How the dentists were selected, how many were surveyed, and what question they were actually asked
Survey results depend critically on sampling method, sample size, and exact question wording — all of which affect the validity and generalizability of the statistic.
Question 7: A company's accident rate dropped from 8% to 4% after mandatory safety training. An analyst notes that the company also hired a new safety officer at the same time. What should the analyst conclude?
- The training program caused the improvement
- The improvement cannot be attributed solely to training because the safety officer's role is a confounding variable (Correct answer)
- The safety officer had no effect on accident rates
- Accident rates always drop over time
Correct answer: The improvement cannot be attributed solely to training because the safety officer's role is a confounding variable
When multiple changes occur simultaneously, isolating the effect of any single variable requires study designs that separate those variables.
Watson-Glaser Critical Thinking Appraisal Interpreting Evidence and Data Questions and Answers 5
A clinical trial reports that a new drug reduced symptoms in 40% of patients compared to 30% on placebo.
The p-value is 0.04.
What is the most accurate interpretation?