NCLB Scientifically Based Research 4 — Questions and Answers
Question 1: A researcher evaluates an after-school math program by comparing pre- and post-test scores for participating students only. Under NCLB, why is this design weak?
- The tests must be administered by an outside agency
- Without a comparison group, gains cannot be attributed to the program rather than maturation or other factors (Correct answer)
- After-school programs are excluded from NCLB's research requirements
- Pre/post designs are prohibited under NCLB
Correct answer: Without a comparison group, gains cannot be attributed to the program rather than maturation or other factors
A design without a comparison group cannot rule out alternative explanations for improvement, failing NCLB's internal validity standard.
Question 2: Under NCLB, 'reliable' data collection instruments are those that:
- Are developed by the U.S. Department of Education
- Produce consistent results when administered under similar conditions (Correct answer)
- Have been used in research for at least 20 years
- Are aligned to state academic standards
Correct answer: Produce consistent results when administered under similar conditions
Reliability means a measurement tool produces consistent, repeatable results across administrations and raters.
Question 3: An educational publisher claims its program is 'research-based.' Under NCLB's stricter definition of scientifically based research, a district administrator should verify that:
- The program has been featured in education trade magazines
- The supporting studies used rigorous experimental or quasi-experimental designs with peer review (Correct answer)
- The publisher has been in business for at least 10 years
- Teacher satisfaction surveys show at least 80% approval
Correct answer: The supporting studies used rigorous experimental or quasi-experimental designs with peer review
NCLB distinguishes 'research-based' (broad) from 'scientifically based research,' which requires rigorous design and peer review.
Question 4: Which aspect of NCLB's scientifically based research definition is addressed by requiring studies to use 'valid' measures?
- Ensuring the measure actually assesses what it claims to measure (Correct answer)
- Ensuring the measure was developed within the last five years
- Ensuring the measure is free of charge to schools
- Ensuring the measure can be completed in under one hour
Correct answer: Ensuring the measure actually assesses what it claims to measure
Validity means the assessment tool genuinely measures the skill or construct it is intended to measure, not something else.
Question 5: NCLB's definition of scientifically based research emphasizes 'systemic, empirical methods.' This phrase means the research must:
- Cover an entire school system from kindergarten through grade 12
- Be conducted using observable, measurable data gathered through disciplined procedures (Correct answer)
- Be funded by a federal grant program
- Involve at least one university partner
Correct answer: Be conducted using observable, measurable data gathered through disciplined procedures
Systemic, empirical methods require that data be observable and measurable, collected through structured, disciplined procedures rather than informal means.
Question 6: Under NCLB, which body is responsible for reviewing programs included in the What Works Clearinghouse to assess their scientifically based research credentials?
- The National Education Association
- The Institute of Education Sciences (IES) (Correct answer)
- The Council of Chief State School Officers
- The National Board for Professional Teaching Standards
Correct answer: The Institute of Education Sciences (IES)
The Institute of Education Sciences (IES) within the U.S. Department of Education oversees the What Works Clearinghouse, which reviews evidence of program effectiveness.
Question 7: A state uses Title II funds for a professional development workshop series. Under NCLB, the state must ensure the workshops are grounded in scientifically based research on:
- School finance and budget management
- Teaching and learning in the relevant content area (Correct answer)
- Community engagement and family involvement only
- Technology integration without regard to learning outcomes
Correct answer: Teaching and learning in the relevant content area
NCLB required Title II professional development to be based on scientifically based research on effective teaching and student learning in the relevant subject area.
NCLB Scientifically Based Research 4
A researcher evaluates an after-school math program by comparing pre- and post-test scores for participating students only.
Under NCLB, why is this design weak?