ORELA Assessment & Evaluation Techniques 5 — Questions and Answers
Question 1: A teacher uses a think-aloud protocol while a student solves math problems. This technique PRIMARILY helps the teacher:
- Calculate a precise score for the student's mathematical fluency
- Understand the student's reasoning process and identify specific misconceptions (Correct answer)
- Compare the student's performance to grade-level benchmarks
- Determine whether the student qualifies for gifted services
Correct answer: Understand the student's reasoning process and identify specific misconceptions
Think-alouds make covert reasoning visible, allowing teachers to pinpoint exactly where and why a student's thinking goes astray.
Question 2: A school team reviews disaggregated assessment data and finds that English Language Learners score significantly lower on a science content test than on hands-on lab performance. This finding MOST strongly suggests the need to:
- Remove ELL students from the accountability reporting system
- Examine whether linguistic complexity in test items is masking students' science knowledge (Correct answer)
- Require ELL students to retake the test with no accommodations
- Lower the passing score on the written test for all students
Correct answer: Examine whether linguistic complexity in test items is masking students' science knowledge
A gap between written-test and hands-on performance for ELLs indicates that language demands of the assessment may be interfering with measurement of science content knowledge.
Question 3: Which of the following scenarios represents a threat to the INTERNAL validity of a classroom action research study?
- The study sample is too small to generalize findings to other schools
- Students in the experimental group also participate in an after-school tutoring program during the study (Correct answer)
- The assessment instrument has not been validated on a national norm group
- The teacher researcher has a positive relationship with students
Correct answer: Students in the experimental group also participate in an after-school tutoring program during the study
Confounding variables, such as simultaneous tutoring, make it impossible to attribute outcome changes solely to the intervention being studied, threatening internal validity.
Question 4: A student receives a grade equivalent score of 5.4 on a reading assessment in third grade. This score MOST accurately means:
- The student should be placed in a fifth-grade reading class immediately
- The student performed as well as an average student in the fourth month of fifth grade would on this test (Correct answer)
- The student has mastered all fifth-grade reading standards
- The student's reading level is two years behind grade level
Correct answer: The student performed as well as an average student in the fourth month of fifth grade would on this test
Grade equivalent scores indicate the grade and month at which the average student in the norming sample earned that same raw score, not that the student has mastered that grade's curriculum.
Question 5: Which practice BEST supports student ownership of learning during the assessment process?
- Providing scores only at the end of a unit to avoid influencing student effort
- Teaching students to self-assess against clear criteria and set specific improvement goals (Correct answer)
- Relying exclusively on teacher-generated feedback without peer input
- Withholding rubrics until after students submit their work
Correct answer: Teaching students to self-assess against clear criteria and set specific improvement goals
When students learn to evaluate their own work against explicit criteria and articulate next steps, they develop metacognitive skills that drive independent improvement.
Question 6: A district wants to ensure that all teachers scoring the same writing assessment assign consistent scores. Which procedure is MOST effective for achieving this goal?
- Having each teacher score only their own students' papers to reduce bias
- Conducting anchor-paper calibration sessions before scoring begins (Correct answer)
- Using a single holistic score rather than an analytic rubric
- Allowing teachers to assign scores based on their individual professional judgment
Correct answer: Conducting anchor-paper calibration sessions before scoring begins
Calibration sessions, where scorers discuss and agree on anchor papers that exemplify each performance level, are the most effective way to build inter-rater reliability.
Question 7: Which of the following is an example of CONVERGENT evidence used to support a validity argument for an assessment?
- Showing that the test predicts future grades AND correlates highly with a well-established measure of the same construct (Correct answer)
- Demonstrating that the test has low correlation with measures of unrelated constructs
- Confirming that all items on the test have similar difficulty levels
- Verifying that the test administration conditions are standardized across sites
Correct answer: Showing that the test predicts future grades AND correlates highly with a well-established measure of the same construct
Convergent validity evidence is established when a new measure correlates positively with other measures of the same construct and with relevant external criteria.
ORELA Assessment & Evaluation Techniques 5
A teacher uses a think-aloud protocol while a student solves math problems.
This technique PRIMARILY helps the teacher: