Bluebook SAT Test Evaluating Statistical Claims: Observational Studies and Experiments 6 — Questions and Answers
Question 1: A university randomly assigns 60 first-year students to peer mentoring and 60 to no mentoring. Those with mentors have higher GPA after one year. A dean applies this to all students at the university. What is the main limitation of this generalization?
- GPAs are too variable to measure across all students.
- The 120 students may not be representative of all first-year students at the university. (Correct answer)
- Random assignment makes the study results impossible to generalize.
- Peer mentoring programs cannot be implemented university-wide.
Correct answer: The 120 students may not be representative of all first-year students at the university.
The causal conclusion is valid within the study, but with only 120 students from potentially non-representative groups, generalizing to all students at the university requires caution.
Question 2: Researchers study whether a new teaching strategy improves reading comprehension by observing classrooms that voluntarily adopt the strategy versus those that do not. What is the primary limitation of this study?
- The sample size is too large for valid comparison.
- Because teachers self-selected the strategy, more motivated or skilled teachers may have adopted it, confounding results. (Correct answer)
- Reading comprehension cannot be measured objectively.
- The study would have been stronger with even more volunteers.
Correct answer: Because teachers self-selected the strategy, more motivated or skilled teachers may have adopted it, confounding results.
Self-selection bias means that teachers who volunteer may be systematically different from those who don't, making it unclear whether the strategy or the teacher quality is responsible for better outcomes.
Question 3: A well-designed randomized experiment conducted on college athletes in one state shows that a protein supplement improves sprint times. A coach in a different state wants to apply this finding to high school students. What concern is most valid?
- Randomized experiments are only valid in laboratory settings.
- High school students may differ from college athletes in age, physiology, and training level, limiting generalizability. (Correct answer)
- The protein supplement would work on all humans regardless of age or fitness level.
- Sprint times cannot be reliably measured in field settings.
Correct answer: High school students may differ from college athletes in age, physiology, and training level, limiting generalizability.
Even though the experiment established causation within the study group, the finding may not generalize to high school students who differ physiologically and in training status from college athletes.
Question 4: An observational study shows that people who drink red wine moderately have fewer cardiovascular events than abstainers. A doctor recommends wine to all patients to protect their heart. Which of the following best explains why this recommendation is premature?
- The study used too few participants.
- The association may be explained by confounders; moderate drinkers may also follow healthier overall lifestyles. (Correct answer)
- Cardiovascular events are rare and cannot be studied in observational data.
- Wine is too inconsistent in quality across studies.
Correct answer: The association may be explained by confounders; moderate drinkers may also follow healthier overall lifestyles.
Moderate drinkers may exercise more, eat better, or have lower baseline stress — confounders that could explain the lower cardiovascular event rate rather than the wine itself.
Question 5: A study randomly assigns 50 households to receive a water filter and 50 households to not receive one. After 6 months, the filter group reports fewer gastrointestinal illnesses. Which of the following is a valid conclusion?
- Using a water filter causes fewer gastrointestinal illnesses in all households worldwide.
- In households similar to those in the study, water filters likely cause fewer gastrointestinal illnesses. (Correct answer)
- The households without filters were negligent about their water supply.
- There is only a correlation between water filters and illness — not a causal link.
Correct answer: In households similar to those in the study, water filters likely cause fewer gastrointestinal illnesses.
Random assignment supports a causal conclusion within the study; however, the result applies to households similar to those studied, not universally.
Question 6: A survey finds that people who own dogs walk more steps per day than non-dog-owners. The researcher concludes: 'Dog ownership increases daily step count.' What type of evidence would strengthen this causal claim?
- A larger survey with a more diverse sample
- A randomized experiment in which some participants are randomly assigned a dog (Correct answer)
- A longitudinal study following dog-owners over 10 years
- A survey in multiple countries confirming the same association
Correct answer: A randomized experiment in which some participants are randomly assigned a dog
Only random assignment to owning or not owning a dog would eliminate confounding (e.g., active people choosing dogs) and allow a causal conclusion.
Question 7: A large randomized experiment with diverse participants from 20 states finds that a new reading intervention improves comprehension scores. A state education board generalizes this to all students in the country. Which concern about generalizability is least valid?
- The study was conducted in only 20 of 50 states.
- The sample may not represent every demographic group equally.
- The study used random assignment, which undermines generalizability. (Correct answer)
- The intervention may work differently in under-resourced schools.
Correct answer: The study used random assignment, which undermines generalizability.
Random assignment has nothing to do with generalizability; it supports causal inference. The legitimate concerns involve sample representativeness and contextual differences.
Bluebook SAT Test Evaluating Statistical Claims: Observational Studies and Experiments 6
A university randomly assigns 60 first-year students to peer mentoring and 60 to no mentoring.
Those with mentors have higher GPA after one year.
A dean applies this to all students at the university.
What is the main limitation of this generalization?