KSA Psychometric Testing & Measurement Theory 2 — Questions and Answers
Question 1: Which Item Response Theory (IRT) model uses only item difficulty to characterize test items?
- 2-parameter logistic model (2PL)
- 3-parameter logistic model (3PL)
- 1-parameter logistic model (Rasch model) (Correct answer)
- Generalizability theory model
Correct answer: 1-parameter logistic model (Rasch model)
The Rasch (1PL) model characterizes items using only the difficulty parameter, assuming all items discriminate equally.
Question 2: Differential Item Functioning (DIF) analysis is conducted primarily to:
- Increase test reliability coefficients
- Identify items that perform differently for subgroups of equal ability (Correct answer)
- Reduce the number of items in a test
- Establish content validity evidence
Correct answer: Identify items that perform differently for subgroups of equal ability
DIF analysis detects items where examinees from different subgroups with the same ability level have systematically different probabilities of responding correctly.
Question 3: A norm-referenced assessment interprets an individual's score by comparing it to:
- A pre-defined performance standard
- The performance of a relevant comparison group (Correct answer)
- The individual's previous test scores
- The content domain coverage
Correct answer: The performance of a relevant comparison group
Norm-referenced assessments rank individuals relative to a normative sample, indicating where a person falls in the distribution of scores.
Question 4: In a criterion-referenced assessment, a cut score is established to:
- Rank candidates from highest to lowest
- Determine who meets a minimum performance standard (Correct answer)
- Identify the mean of the score distribution
- Measure inter-rater reliability
Correct answer: Determine who meets a minimum performance standard
A cut score defines the minimum level of performance required to be classified as competent or passing on a criterion-referenced assessment.
Question 5: Which method of setting a cut score asks subject matter experts to estimate the probability that a minimally competent candidate would answer each item correctly?
- Angoff method (Correct answer)
- Bookmark method
- Contrasting groups method
- Body of Work method
Correct answer: Angoff method
The Angoff method has SMEs estimate the probability that a minimally competent examinee would answer each item correctly; these probabilities are averaged to set the cut score.
Question 6: Generalizability theory (G-theory) extends classical test theory primarily by:
- Replacing reliability with validity coefficients
- Partitioning observed score variance into multiple sources simultaneously (Correct answer)
- Eliminating the need for a normative sample
- Requiring IRT parameter estimation
Correct answer: Partitioning observed score variance into multiple sources simultaneously
G-theory allows researchers to identify and quantify multiple sources of measurement error (e.g., raters, occasions, items) simultaneously using an ANOVA framework.
Question 7: When a KSA test has high construct validity, it means the test:
- Covers all content domains proportionally
- Accurately measures the theoretical trait or construct it purports to assess (Correct answer)
- Produces consistent scores across administrations
- Predicts future job performance well
Correct answer: Accurately measures the theoretical trait or construct it purports to assess
Construct validity provides evidence that the assessment operationalizes and measures the intended psychological or behavioral construct.
KSA Psychometric Testing & Measurement Theory 2
Which Item Response Theory (IRT) model uses only item difficulty to characterize test items?