ATP Case Studies & Practical Application 3 — Questions and Answers
Question 1: A testing company receives a subpoena requesting examinee score records and test booklets for a legal case. What is the first step the company should take?
- Immediately comply with the subpoena and release all requested materials
- Consult legal counsel to evaluate the scope of the subpoena and determine what can be released under applicable confidentiality agreements and test security requirements (Correct answer)
- Destroy the records to protect examinee privacy
- Contact the examinee directly and ask for permission before responding
Correct answer: Consult legal counsel to evaluate the scope of the subpoena and determine what can be released under applicable confidentiality agreements and test security requirements
Legal counsel must review any subpoena to balance compliance obligations with confidentiality agreements, test security, and relevant state and federal privacy laws.
Question 2: During a validity study for a police officer selection test, researchers find the test predicts performance well overall but shows differential prediction for female officers. What is the correct psychometric response?
- Drop female officers from the validity study sample
- Conduct separate regression analyses by group and examine whether differential slopes or intercepts indicate biased prediction (Correct answer)
- Use a single regression line and ignore group differences
- Increase the cut score for all applicants to reduce the effect
Correct answer: Conduct separate regression analyses by group and examine whether differential slopes or intercepts indicate biased prediction
Differential prediction analysis using separate regression lines by subgroup is the standard method for detecting and reporting potential test bias in predictive validity studies.
Question 3: A state department of education wants to use a vendor's nationally normed achievement test for statewide accountability. What is the primary validity concern?
- Whether the test is long enough to cover the school year
- Whether the test's content aligns with state academic standards and whether national norms are appropriate for state accountability inferences (Correct answer)
- Whether the test is available in multiple languages
- Whether the vendor has experience working with state agencies
Correct answer: Whether the test's content aligns with state academic standards and whether national norms are appropriate for state accountability inferences
Content alignment with state standards is essential for accountability validity, and national norms may not reflect state-specific performance expectations.
Question 4: A test security investigator identifies a cluster of examinees with near-identical incorrect responses in a proctored testing center. What does this pattern most likely indicate?
- All examinees studied from the same preparation materials
- Potential proxy testing or collusion, warranting further forensic analysis including seat proximity and registration data review (Correct answer)
- Random chance patterns that should not be investigated
- A systematic scoring error by the test vendor
Correct answer: Potential proxy testing or collusion, warranting further forensic analysis including seat proximity and registration data review
Identical incorrect response patterns (answer copying indices) in a specific testing center are a recognized statistical signal of potential collusion or proxy testing requiring forensic follow-up.
Question 5: A medical licensing board wants to raise its passing standard to address public safety concerns. Which approach best balances rigor with defensibility?
- Set a higher arbitrary score based on board members' intuition
- Convene a new standard-setting panel using a recognized method (e.g., modified Angoff or Bookmark), document the process, and validate the new standard against competency evidence (Correct answer)
- Automatically fail the bottom 20% of candidates
- Commission an external audit and adopt whatever the auditor recommends without further review
Correct answer: Convene a new standard-setting panel using a recognized method (e.g., modified Angoff or Bookmark), document the process, and validate the new standard against competency evidence
Standard-setting with a recognized method, expert panelists, and documented validation is the professionally and legally defensible way to revise a passing standard.
Question 6: An online testing platform reports a significant drop in average scores across all forms after a software update. What is the most likely first hypothesis to investigate?
- Examinees have become less prepared over the past month
- A software or interface change may have introduced construct-irrelevant variance, such as altered timing, display issues, or navigation changes (Correct answer)
- The test forms released after the update were inherently harder
- Random statistical fluctuation that will correct itself next month
Correct answer: A software or interface change may have introduced construct-irrelevant variance, such as altered timing, display issues, or navigation changes
Sudden cross-form score drops following a software update most likely indicate a platform-level construct-irrelevant factor introduced by the update.
Question 7: A certification exam uses a fixed cut score, but a policy analyst argues it should be adjusted annually based on pass rates. What is the strongest argument against annual cut score adjustments based on pass rates?
- It would require updating the test blueprint each year
- Adjusting the cut score based on pass rates conflates norm-referenced and criterion-referenced interpretations, undermining the standard's meaning as a fixed competency threshold (Correct answer)
- It would make the exam more expensive to administer
- Examinees would appeal more frequently if the standard changed annually
Correct answer: Adjusting the cut score based on pass rates conflates norm-referenced and criterion-referenced interpretations, undermining the standard's meaning as a fixed competency threshold
A criterion-referenced cut score represents a fixed competency standard; adjusting it to achieve a target pass rate converts it into a norm-referenced selection mechanism, invalidating its criterion-referenced meaning.
ATP Case Studies & Practical Application 3
A testing company receives a subpoena requesting examinee score records and test booklets for a legal case.
What is the first step the company should take?