Test Validation and Reliability Flashcards

Which type of validity evidence examines whether test scores correlate with an external criterion measured at the same time?

Answer: Concurrent validity

Concurrent validity is established when test scores correlate with a criterion measure collected simultaneously.

A reliability coefficient of 0.50 for a selection test would generally be considered:

Answer: Too low for most selection purposes

A reliability coefficient of 0.50 is generally considered too low for employment selection tests, which typically require 0.70 or higher.

What does the standard error of measurement (SEM) primarily tell us?

Answer: The expected variation in an individual's scores across repeated administrations

SEM estimates how much an individual's observed score is likely to vary if the test were administered multiple times.

When a test is used to predict future job performance measured six months after hiring, this is an example of:

Answer: Predictive validity

Predictive validity involves collecting criterion data after a time delay following test administration.

Which reliability method requires only a single test administration?

Answer: Internal consistency reliability

Internal consistency methods (e.g., Cronbach's alpha, KR-20) assess reliability using data from a single test administration.

Differential item functioning (DIF) analysis is used to identify items that:

Answer: Perform differently for subgroups after matching on ability

DIF flags items that favor or disadvantage a particular subgroup after controlling for overall ability level.

In classical test theory, a person's observed score is defined as:

Answer: True score plus measurement error

Classical test theory states that Observed Score = True Score + Error Score.

A job analysis identifies five critical task clusters for a position. A test that covers only two of those clusters likely lacks:

Answer: Content validity

Content validity requires adequate coverage of all critical KSAs and tasks identified in the job analysis.

The attenuation of a validity coefficient due to restriction of range refers to:

Answer: A coefficient that is artificially deflated because the sample lacks score variability

Range restriction occurs when hired employees represent a narrow band of scores, which deflates observed validity coefficients.

Which statistic is most commonly used to assess internal consistency reliability for tests with polytomous (multi-point) items?

Answer: Cronbach's alpha

Cronbach's alpha is appropriate for items with multiple response categories, while KR-20 is for dichotomous items.

A validity generalization (VG) study concludes that a test has consistent validity across multiple settings. This finding supports:

Answer: The transportability of test validity to new settings without local studies

Validity generalization supports using existing validity evidence to infer a test's validity in a new setting without conducting a full local study.

When evaluating construct validity, convergent evidence is demonstrated when:

Answer: The test correlates highly with measures of theoretically similar constructs

Convergent evidence shows that measures of the same or related constructs correlate substantially with each other.

The Uniform Guidelines on Employee Selection Procedures recommend that a content validity strategy is most defensible when the selection procedure measures:

Answer: Observable work behaviors or knowledge used on the job

The Uniform Guidelines indicate content validity is most appropriate for tests measuring observable work behaviors or directly job-related knowledge.

Inter-rater reliability is MOST critical to establish when:

Answer: Structured interviews or work samples involve human judgment in scoring

Inter-rater reliability ensures that different evaluators assign consistent scores when human judgment is involved in the scoring process.

A test developer applies the Spearman-Brown prophecy formula. What is the primary purpose of this formula?

Answer: To estimate the reliability of a lengthened or shortened test

The Spearman-Brown formula predicts how reliability will change as a function of increasing or decreasing test length.

Which scenario best illustrates the problem of criterion contamination?

Answer: Supervisors know an employee's test score before providing performance ratings

Criterion contamination occurs when supervisors' knowledge of test scores influences their performance ratings, biasing the criterion.

In item response theory (IRT), which parameter describes how steeply an item distinguishes between higher and lower ability test takers?

Answer: The a-parameter (discrimination)

The a-parameter in IRT reflects item discrimination — how sharply the item separates individuals with different ability levels.

A selection test shows high validity for a sample of incumbents but poor validity for a sample of applicants. The MOST likely explanation is:

Answer: Range restriction in the incumbent sample inflated the coefficient

Incumbent samples are often range-restricted because low performers have already left, which can inflate validity coefficients relative to applicant samples.

Consequential validity refers to the idea that validation should include evidence about:

Answer: The intended and unintended social consequences of test use

Consequential validity (Messick's framework) incorporates evidence about the ethical and social impact of testing decisions.

When a test battery includes both a cognitive ability test and a structured interview, and the two are combined into a total score, this practice is known as:

Answer: Compensatory combination

Compensatory combination allows high scores on one predictor to offset low scores on another when computing a composite score.