Test Validity and Reliability Flashcards

The concept of 'validity generalization' applied to TAPAS suggests that:

Answer: Validity evidence accumulated across settings supports use in new, similar contexts

Validity generalization (Schmidt & Hunter) shows that observed variation in validity coefficients across studies is largely due to statistical artifacts, allowing evidence from prior studies to support new applications.

Range restriction in TAPAS validity studies most commonly occurs because:

Answer: Only individuals who scored high enough to be selected are included in the criterion sample

When only selected (high-scoring) applicants enter the criterion sample, the score distribution is truncated, which attenuates observed validity coefficients relative to the true population value.

A TAPAS score that is highly reliable but measures the wrong construct exemplifies:

Answer: The distinction that reliability is necessary but not sufficient for validity

Reliability is a prerequisite for validity—a score must be consistent before it can validly measure anything—but consistent measurement of the wrong thing is reliable yet invalid.

When conducting a TAPAS item review panel to establish content validity, the panel should ideally consist of:

Answer: Subject-matter experts familiar with the construct and the target population

Content validity panels require subject-matter experts who understand both the psychological construct (e.g., conscientiousness) and the job context (military service) to judge item representativeness and relevance.

In a TAPAS validity study, the disattenuation formula is applied to correct an observed correlation for:

Answer: Measurement error in both the predictor and criterion

The disattenuation (correction for attenuation) formula adjusts an observed correlation upward to estimate the true-score correlation by removing the dampening effect of measurement error in both variables.

Which scenario would provide the strongest evidence that TAPAS scores have utility beyond selection and should inform personnel classification decisions?

Answer: Different TAPAS profiles predict success differentially across distinct military job families

If specific personality profiles predict success in some jobs better than others (differential prediction), TAPAS can improve classification by matching individuals to the roles where they are most likely to thrive.

A cross-validation study of TAPAS shrinks the initial validity coefficient considerably when applied to a new sample. This shrinkage is best explained by:

Answer: Capitalization on chance in the original model, which does not replicate

Shrinkage occurs because a model optimized on one sample fits sample-specific noise; cross-validation reveals the true, lower validity by applying the model to new data where that noise is absent.