CBT Adaptive Testing and Computer Adaptive Testing (CAT) 2 — Questions and Answers
Question 1: The ability estimate (theta) in a CAT is updated after each item response using:
- Raw score averaging
- Maximum Likelihood Estimation (MLE) or Bayesian estimation (Correct answer)
- Percentile rank conversion
- Standard score equating
Correct answer: Maximum Likelihood Estimation (MLE) or Bayesian estimation
CAT systems iteratively update theta using MLE or Bayesian methods (such as EAP or MAP) after each item response to refine the ability estimate.
Question 2: Shadow testing in CAT refers to:
- Administering duplicate items to detect cheating
- Constructing a complete ghost test that satisfies all constraints before selecting the next item (Correct answer)
- Logging item responses for post-test review
- Delivering a practice test before the operational CAT
Correct answer: Constructing a complete ghost test that satisfies all constraints before selecting the next item
Shadow testing builds a full, constraint-satisfying test in the background at each step and selects the next item from that shadow test, ensuring all blueprint and exposure constraints will be met.
Question 3: A CAT that uses a three-parameter logistic (3PL) IRT model accounts for which additional parameter compared to the 1PL (Rasch) model?
- Item discrimination only
- Item discrimination and guessing (pseudo-chance) (Correct answer)
- Item time limit and word count
- Partial credit scoring
Correct answer: Item discrimination and guessing (pseudo-chance)
The 3PL model adds discrimination (a) and pseudo-guessing (c) parameters to the difficulty (b) parameter of the Rasch model, accounting for the probability of correct response by low-ability examinees through guessing.
Question 4: Early termination rules in mastery-based CAT (e.g., licensure tests) are triggered when:
- The examinee's time limit is reached
- The algorithm is sufficiently confident the examinee is above or below the passing standard (Correct answer)
- The item bank is exhausted
- The examinee requests to end the test
Correct answer: The algorithm is sufficiently confident the examinee is above or below the passing standard
Mastery CATs use sequential probability ratio tests or similar decision rules to stop early when the probability of misclassifying the examinee falls below an acceptable error rate.
Question 5: Which of the following best describes item parameter drift in the context of CAT?
- Changes in examinee motivation across administrations
- A change in an item's statistical properties over time due to repeated exposure or population shifts (Correct answer)
- The gradual increase in item bank size
- Variation in item difficulty caused by translation
Correct answer: A change in an item's statistical properties over time due to repeated exposure or population shifts
Item parameter drift occurs when an item's IRT parameters (b, a, c) change from their calibrated values, often because the item has been exposed and memorized or because the examinee population has shifted.
Question 6: A CAT item bank is considered adequate when it:
- Contains exactly 100 items per content area
- Has sufficient items across the ability continuum with controlled exposure rates to support reliable adaptive administration (Correct answer)
- Includes only dichotomously scored items
- Is updated no more than once per year
Correct answer: Has sufficient items across the ability continuum with controlled exposure rates to support reliable adaptive administration
An adequate CAT item bank must have enough items at varying difficulty levels, across all content domains, and in sufficient quantity so that exposure limits can be enforced without depleting available items.
Question 7: In CAT, the concept of 'theta initialization' refers to:
- Setting the passing score before the test begins
- Choosing the starting ability estimate for the first item selection (Correct answer)
- Calibrating item parameters prior to operational use
- Fixing the test length before administration
Correct answer: Choosing the starting ability estimate for the first item selection
Theta initialization sets the assumed starting ability level (often at the mean or a neutral value) used to select the first item before any examinee response data is available.
CBT Adaptive Testing and Computer Adaptive Testing (CAT) 2
The ability estimate (theta) in a CAT is updated after each item response using: