CBT Psychometric Principles and Analysis 5 — Questions and Answers
Question 1: Which of the following is the primary advantage of polytomous IRT models (e.g., Graded Response Model) over dichotomous models?
- They require smaller sample sizes for calibration
- They model ordered response categories, capturing more score information per item (Correct answer)
- They eliminate the need for item discrimination parameters
- They are immune to guessing effects
Correct answer: They model ordered response categories, capturing more score information per item
Polytomous models explicitly account for multiple ordered response options (e.g., Likert scales), extracting more information from each item than a simple pass/fail scoring.
Question 2: Which statistic is most appropriate for estimating reliability when a test has items scored on multiple response categories (e.g., 0–3 points each)?
- Kuder-Richardson Formula 20 (KR-20)
- Split-half reliability with Spearman-Brown correction
- Coefficient alpha (Cronbach's alpha) (Correct answer)
- Test-retest reliability
Correct answer: Coefficient alpha (Cronbach's alpha)
KR-20 is limited to dichotomous items; Cronbach's alpha is the appropriate generalization for items with multiple score points or continuous responses.
Question 3: In a multitrait-multimethod (MTMM) matrix, convergent validity is supported when:
- Correlations between different traits measured by the same method are high
- Correlations between the same trait measured by different methods are high (Correct answer)
- All correlations in the matrix are statistically non-significant
- Method factors explain more variance than trait factors
Correct answer: Correlations between the same trait measured by different methods are high
Convergent validity in MTMM is evidenced by high correlations in the validity diagonal—same trait, different methods—indicating the construct is being measured consistently.
Question 4: A test developer wants to assess whether scores predict job performance 6 months after hiring. Which validity study design should be used?
- Concurrent criterion-related validity study
- Predictive criterion-related validity study (Correct answer)
- Content validity review panel
- Known-groups validity study
Correct answer: Predictive criterion-related validity study
Predictive validity studies collect test scores first, then measure criterion performance after a time interval, directly evaluating the test's forecasting utility.
Question 5: Which of the following best defines 'construct underrepresentation' as a threat to validity?
- The test includes items that measure constructs outside the intended domain
- The test fails to capture important facets of the intended construct (Correct answer)
- Examinees misunderstand item wording due to cultural differences
- The scoring rubric allows excessive rater subjectivity
Correct answer: The test fails to capture important facets of the intended construct
Construct underrepresentation occurs when the test is too narrow and omits important aspects of the construct, limiting the generalizability of score interpretations.
Question 6: Which of the following is a key feature that distinguishes norm-referenced score interpretation from criterion-referenced interpretation?
- Norm-referenced tests always use multiple-choice items
- Norm-referenced interpretation requires comparing an individual's score to a standardization sample (Correct answer)
- Criterion-referenced tests cannot be used for high-stakes decisions
- Norm-referenced tests never report raw scores
Correct answer: Norm-referenced interpretation requires comparing an individual's score to a standardization sample
Norm-referenced interpretation derives meaning by comparing an individual's score to the distribution of scores from a defined reference group (norm sample).
Question 7: In item analysis, a distractor analysis reveals that for a 4-option item, Option C (wrong) is chosen by 45% of low-scoring examinees and only 5% of high-scoring examinees. What does this indicate about Option C?
- Option C is a poor distractor and should be revised
- Option C is a functioning distractor that effectively discriminates ability levels (Correct answer)
- Option C is ambiguous and should be treated as correct
- Option C should be removed to simplify the item
Correct answer: Option C is a functioning distractor that effectively discriminates ability levels
An effective distractor attracts low-ability examinees but is rarely chosen by high-ability examinees; this pattern confirms Option C is performing as intended.
CBT Psychometric Principles and Analysis 5
Which of the following is the primary advantage of polytomous IRT models (e.g., Graded Response Model) over dichotomous models?