ATP Research & Evidence-Based Practice 5

Question 1

Which approach to validity argument, articulated by Messick, emphasizes that validity is a unitary concept focused on:

Accepted Answer

The appropriateness of inferences and actions based on test scores

Answer

Messick's unified conception holds that validity is a single integrated judgment about the degree to which evidence supports the intended interpretations and uses of test scores.

Question 2

A test developer discovers that a high-stakes certification exam has a significant adverse impact ratio below 4/5 for one ethnic group. The NEXT recommended step is to:

Accepted Answer

Investigate whether the score differences reflect construct-relevant or irrelevant factors

Answer

The 4/5 rule triggers further investigation, not automatic action; the critical question is whether group differences stem from real construct-related factors or test bias.

Question 3

Which reliability approach is most appropriate for assessing inter-rater agreement on a constructed-response item with ordinal scoring rubrics?

Accepted Answer

Weighted kappa

Answer

Weighted kappa accounts for ordinal scoring by penalizing disagreements proportionally to their distance, making it more appropriate than simple kappa or Pearson r for rubric-scored responses.

Question 4

Generalizability theory extends classical test theory by partitioning score variance into multiple facets, such as:

Accepted Answer

Person, item, rater, and occasion effects

Answer

G-theory designs allow simultaneous estimation of variance components attributable to persons, items, raters, occasions, and their interactions.

Question 5

A licensing board wants to set a defensible passing score. Which of the following best strengthens the legal defensibility of the standard-setting process?

Accepted Answer

Documenting the qualifications of panelists and the procedural steps followed

Answer

Legal defensibility depends on transparent documentation of panelist qualifications, training, procedures, and rationale, demonstrating due diligence in the standard-setting process.

ATP - Association of Test Publishers Practice Test

ATP - Association of Test Publishers Practice Test

ATP Research & Evidence-Based Practice 5