ATP Case Studies & Practical Application 2 — Questions and Answers
Question 1: A testing organization discovers mid-administration that a subset of test items had a technical error causing ambiguous wording. What is the most appropriate immediate response?
- Cancel the entire test administration and reschedule all examinees
- Flag the affected items for post-hoc analysis and consider score adjustments or item removal (Correct answer)
- Ignore the issue and score all items as written
- Announce the error to examinees during the test to allow correction
Correct answer: Flag the affected items for post-hoc analysis and consider score adjustments or item removal
Flagging affected items for post-hoc analysis allows the psychometric team to evaluate impact and apply appropriate scoring adjustments while minimizing disruption.
Question 2: A credentialing body finds that its passing standard was set without formal job task analysis. Which remediation step is most critical to ensure defensibility?
- Retroactively lower the cut score to pass more candidates
- Commission a new job task analysis and conduct a standard-setting study to establish a defensible cut score (Correct answer)
- Keep the existing standard because changing it would create legal risk
- Ask subject matter experts to vote on whether the current standard seems reasonable
Correct answer: Commission a new job task analysis and conduct a standard-setting study to establish a defensible cut score
A new job task analysis followed by a formal standard-setting study establishes the empirical and content linkage required for defensible credentialing decisions.
Question 3: An employer uses a cognitive ability test that shows adverse impact against a protected group. Under the Uniform Guidelines, which action best addresses this situation?
- Discontinue use of the test immediately without further analysis
- Conduct a validity study to determine whether the test predicts job performance and explore alternative tests with less adverse impact (Correct answer)
- Lower the passing score for the affected group only
- Increase the sample size of the protected group in the next administration
Correct answer: Conduct a validity study to determine whether the test predicts job performance and explore alternative tests with less adverse impact
The Uniform Guidelines require employers to demonstrate validity for any selection procedure causing adverse impact or to seek less adverse alternatives.
Question 4: A test developer is adapting an English-language licensure exam for Spanish-speaking candidates. Which process is most critical to ensure linguistic equivalence?
- Translate the test using a single bilingual expert and administer it immediately
- Perform forward-and-back translation, conduct bilingual review panels, and run pilot studies to verify item performance (Correct answer)
- Use automated translation software and compare scores statistically
- Have examinees choose their preferred language on the day of testing without any prior adaptation process
Correct answer: Perform forward-and-back translation, conduct bilingual review panels, and run pilot studies to verify item performance
Forward-and-back translation with bilingual expert review and pilot testing is the standard methodology for ensuring linguistic and construct equivalence in test adaptation.
Question 5: A large-scale K-12 assessment program is under scrutiny because schools are 'teaching to the test.' What is the most appropriate psychometric and policy response?
- Eliminate the assessment program to stop the practice
- Distinguish between coaching on the construct versus coaching on specific items, and periodically refresh item pools to maintain construct validity (Correct answer)
- Increase test security so teachers cannot see items
- Report only school-level scores to reduce pressure
Correct answer: Distinguish between coaching on the construct versus coaching on specific items, and periodically refresh item pools to maintain construct validity
Differentiating construct-relevant preparation from item-specific coaching, combined with item pool renewal, protects validity while acknowledging that legitimate test preparation is acceptable.
Question 6: A professional certification exam is challenged in court because the test blueprint was developed without practicing professionals' input. Which standard is most relevant to this challenge?
- Standards for Fairness in Testing
- Standards for Educational and Psychological Testing — content validity requirements (Correct answer)
- ADA accommodation standards
- FERPA disclosure requirements
Correct answer: Standards for Educational and Psychological Testing — content validity requirements
The Standards for Educational and Psychological Testing require that content validity evidence be based on a systematic analysis of the job or domain, including input from practicing professionals.
Question 7: An organization administers both a paper-and-pencil and a computer-based version of the same certification exam. Examinees performing better on the CBT version raises concern. What analysis should be conducted first?
- Immediately discontinue the paper version to eliminate the discrepancy
- Conduct differential item functioning (DIF) and mode-equivalence studies to determine whether the score scales are comparable (Correct answer)
- Assume CBT is more valid and adjust paper scores upward
- Survey examinees about their technology comfort levels
Correct answer: Conduct differential item functioning (DIF) and mode-equivalence studies to determine whether the score scales are comparable
DIF and mode-equivalence studies are the standard approach to determining whether test scores from different delivery modes can be placed on a common scale.
ATP Case Studies & Practical Application 2
A testing organization discovers mid-administration that a subset of test items had a technical error causing ambiguous wording.
What is the most appropriate immediate response?