ORELA Assessment & Evaluation Techniques 2 — Questions and Answers
Question 1: A teacher notices that students from one demographic group consistently score lower on a multiple-choice test than on performance tasks measuring the same skills. This discrepancy most likely indicates:
- Construct irrelevant variance due to cultural or linguistic bias in the test items (Correct answer)
- The performance tasks are too easy for the lower-scoring group
- Multiple-choice tests are inherently more valid than performance tasks
- The students need remediation regardless of performance task scores
Correct answer: Construct irrelevant variance due to cultural or linguistic bias in the test items
When scores differ by group on the same construct, construct irrelevant variance—often from biased item language or cultural assumptions—is the most likely culprit.
Question 2: Which of the following best describes the purpose of an item analysis after administering a classroom test?
- To determine the overall difficulty of the test for grading purposes
- To identify individual items that may be flawed or that discriminate poorly between high and low performers (Correct answer)
- To calculate the final grades for all students in the class
- To compare the class average against a national norm group
Correct answer: To identify individual items that may be flawed or that discriminate poorly between high and low performers
Item analysis examines difficulty indices and discrimination indices to flag items that may be misleading, too easy, too hard, or failing to differentiate student understanding.
Question 3: A district uses the same reading assessment each fall for three consecutive years to track cohort growth. What threat to validity should the assessment team be most concerned about?
- Regression to the mean affecting only the highest scorers
- Practice effects and item exposure reducing the assessment's construct validity over time (Correct answer)
- Insufficient sample size in the first year of administration
- The test being too long for third-year administration
Correct answer: Practice effects and item exposure reducing the assessment's construct validity over time
Repeated use of identical items allows students to benefit from prior exposure, threatening validity because scores may reflect familiarity rather than growth in the target skill.
Question 4: In standards-based grading, a student's final grade should primarily reflect:
- Averaged scores from all assignments throughout the grading period
- The most consistent and recent evidence of mastery of learning targets (Correct answer)
- Participation, effort, and homework completion weighted equally with tests
- A single high-stakes end-of-unit exam
Correct answer: The most consistent and recent evidence of mastery of learning targets
Standards-based grading emphasizes reporting the level of mastery a student has demonstrated, prioritizing recent and consistent evidence over early or isolated scores.
Question 5: Which assessment approach is MOST appropriate when the goal is to evaluate a student's ability to apply knowledge to novel, real-world problems?
- Selected-response items with four answer choices
- Curriculum-based measurement probes administered weekly
- Authentic performance tasks requiring demonstration of competencies (Correct answer)
- True/false questions covering declarative knowledge
Correct answer: Authentic performance tasks requiring demonstration of competencies
Authentic performance tasks require students to apply skills in realistic contexts, making them the most appropriate format for assessing transfer and application.
Question 6: A teacher records observational notes about student discussion contributions each day over two weeks. This approach is BEST described as:
- Summative assessment used to assign end-of-unit grades
- Anecdotal record-keeping as a form of formative assessment (Correct answer)
- High-stakes standardized assessment of oral communication
- Norm-referenced evaluation of speaking skills
Correct answer: Anecdotal record-keeping as a form of formative assessment
Systematic anecdotal records gathered during instruction constitute formative assessment that informs ongoing instructional decisions.
Question 7: When interpreting a student's score on a norm-referenced test, a percentile rank of 72 means:
- The student answered 72% of the questions correctly
- The student scored higher than 72% of the students in the norm group (Correct answer)
- The student's score is 72 points above the national average
- The student is performing at the 72nd grade-equivalent level
Correct answer: The student scored higher than 72% of the students in the norm group
A percentile rank indicates the percentage of students in the norm group who scored at or below the student's score, not the percentage of items answered correctly.
ORELA Assessment & Evaluation Techniques 2
A teacher notices that students from one demographic group consistently score lower on a multiple-choice test than on performance tasks measuring the same skills.
This discrepancy most likely indicates: