ATP Quality Control & Assurance 3 — Questions and Answers
Question 1: Which type of reliability coefficient is most appropriate for evaluating consistency of scores on a single administration of a multiple-choice test?
- Test-retest reliability
- Parallel-forms reliability
- Internal consistency (e.g., coefficient alpha) (Correct answer)
- Inter-rater reliability
Correct answer: Internal consistency (e.g., coefficient alpha)
Internal consistency coefficients like Cronbach's alpha or KR-20 measure how consistently items measure the same construct within a single administration.
Question 2: A testing organization discovers a subset of items has been exposed on a social media site prior to the administration. Which quality control step should have prevented this?
- Differential item functioning screening
- Pre-equating using anchor items
- Secure item banking with access controls (Correct answer)
- Post-hoc item exposure rate analysis
Correct answer: Secure item banking with access controls
Robust access controls on the item bank, including role-based permissions and audit logs, are designed to prevent unauthorized disclosure of live items.
Question 3: Which validity evidence source involves examining whether test scores predict relevant future outcomes such as job performance?
- Content validity evidence
- Internal structure evidence
- Criterion-related (predictive) validity evidence (Correct answer)
- Convergent validity evidence
Correct answer: Criterion-related (predictive) validity evidence
Predictive validity evidence demonstrates that scores are meaningfully related to a criterion measured at a later time, supporting score interpretations for selection or licensure.
Question 4: When conducting a standard-setting study, which practice supports quality assurance of panelist judgments?
- Using only content experts with no assessment training
- Providing panelists with impact data and allowing discussion between rounds (Correct answer)
- Instructing panelists to match the current cut score
- Limiting the study to a single round of ratings
Correct answer: Providing panelists with impact data and allowing discussion between rounds
Sharing impact data and facilitating structured discussion between rounds (as in Angoff or bookmark methods) helps panelists calibrate their judgments and reach defensible consensus.
Question 5: A QA checklist for item writing includes verifying that stems avoid 'window dressing.' What does this term mean?
- Using overly technical vocabulary
- Including irrelevant information that does not affect the answer (Correct answer)
- Presenting items in a visually complex format
- Using negatively worded stems
Correct answer: Including irrelevant information that does not affect the answer
Window dressing refers to extraneous detail in the item stem that may confuse examinees or artificially inflate item difficulty without adding construct-relevant information.
Question 6: In computer-adaptive testing (CAT), which quality control parameter limits the overexposure of specific items?
- Maximum Fisher information criterion
- Item exposure rate cap (Correct answer)
- Minimum item bank size requirement
- Adaptive routing threshold
Correct answer: Item exposure rate cap
Item exposure rate caps set a maximum frequency at which any single item can be administered, protecting test security while maintaining measurement precision.
Question 7: According to ATP standards, what must a technical manual for a credentialing exam include?
- Marketing claims about exam validity
- Psychometric evidence supporting score interpretation and use (Correct answer)
- Examinee performance by individual testing site
- Vendor pricing for item development
Correct answer: Psychometric evidence supporting score interpretation and use
Technical manuals must document reliability, validity evidence, scaling, equating, and standard-setting procedures so stakeholders can evaluate the defensibility of score use.
ATP Quality Control & Assurance 3
Which type of reliability coefficient is most appropriate for evaluating consistency of scores on a single administration of a multiple-choice test?