ATP Case Studies & Practical Application 4 — Questions and Answers
Question 1: A test publisher is developing a new workplace integrity test and wants to validate it. Which validation strategy provides the strongest evidence for use in pre-employment screening?
- Content validity only, based on expert review of item content
- A predictive validity study correlating pre-hire scores with post-hire counterproductive work behavior outcomes (Correct answer)
- Concurrent validity using current employees who self-report their integrity levels
- Face validity ratings by HR managers who review the items
Correct answer: A predictive validity study correlating pre-hire scores with post-hire counterproductive work behavior outcomes
Predictive validity using actual hire-and-follow-up designs provides the strongest criterion-related validity evidence for employment selection instruments.
Question 2: A high-stakes exam vendor discovers that a prep company has reconstructed approximately 30% of the live item pool through systematic examinee debriefing. What is the most appropriate response?
- File a public complaint on social media to warn the testing community
- Retire the compromised items immediately, build replacement items, investigate legal remedies under copyright law, and review exam security protocols (Correct answer)
- Continue using the compromised items because retiring them would disadvantage recent examinees
- Lower the passing score to compensate for inflated scores
Correct answer: Retire the compromised items immediately, build replacement items, investigate legal remedies under copyright law, and review exam security protocols
Immediate item retirement, replacement, legal action, and security review are the required steps to restore exam integrity when significant item exposure is confirmed.
Question 3: An examinee with a documented anxiety disorder requests extended time as a testing accommodation. The testing organization must balance accommodation validity with fairness. Which principle guides this decision?
- Accommodations should always be denied to maintain standardization
- Accommodations should remove barriers created by the disability without altering the construct being measured (Correct answer)
- Extended time should be granted to all examinees to avoid discrimination claims
- The organization should rely solely on the examinee's self-report without documentation
Correct answer: Accommodations should remove barriers created by the disability without altering the construct being measured
The purpose of accommodations is to provide access by removing disability-related barriers while preserving the integrity of the construct the test measures.
Question 4: A test developer notices that item difficulty values from the field test are substantially higher (easier) than expected based on expert difficulty estimates. What is the most likely explanation?
- The field test sample was unrepresentative of the target examinee population, skewing toward higher-ability individuals (Correct answer)
- The items were too long
- Field test conditions were not standardized
- Expert estimates are always more accurate than empirical data
Correct answer: The field test sample was unrepresentative of the target examinee population, skewing toward higher-ability individuals
If field test p-values are systematically higher than expert predictions, the field test sample likely overrepresents higher-ability examinees relative to the target population.
Question 5: A credentialing organization wants to implement continuous testing (year-round administration) with a large rotating item pool. What is the primary psychometric challenge this introduces?
- Examinees will study for longer periods
- Maintaining score comparability across different item sets and administration windows requires robust equating and item parameter stability over time (Correct answer)
- The registration process becomes more complex
- Proctors must be trained more frequently
Correct answer: Maintaining score comparability across different item sets and administration windows requires robust equating and item parameter stability over time
Continuous testing with rotating item banks requires ongoing equating to ensure that scores from different forms and windows remain on a common, comparable scale.
Question 6: After an accessibility audit, a testing organization learns that its online exam interface fails WCAG 2.1 AA standards for screen-reader compatibility. What is the legally and ethically appropriate response?
- Refer all screen-reader users to the paper-and-pencil version indefinitely
- Remediate the interface to meet WCAG 2.1 AA, provide interim accommodations for affected examinees, and establish a timeline for full compliance (Correct answer)
- Argue that screen-reader compatibility is not required for certification exams
- Add a disclaimer that the exam is not designed for assistive technology users
Correct answer: Remediate the interface to meet WCAG 2.1 AA, provide interim accommodations for affected examinees, and establish a timeline for full compliance
Testing organizations have legal obligations under the ADA and ethical obligations under professional standards to ensure accessible test delivery; remediation with interim accommodations is the appropriate response.
Question 7: A testing company is asked by a client to release item-level response data to a researcher studying test-taking strategies. What must the company consider before releasing the data?
- Whether the researcher has a PhD in psychometrics
- Whether disclosure would compromise item security, whether examinee confidentiality can be maintained through de-identification, and whether appropriate data use agreements are in place (Correct answer)
- Whether the client has already paid for the research study
- Whether the researcher works for a competing test publisher
Correct answer: Whether disclosure would compromise item security, whether examinee confidentiality can be maintained through de-identification, and whether appropriate data use agreements are in place
Releasing item-level data requires balancing research value against item security and examinee privacy, governed by data use agreements and de-identification protocols.
ATP Case Studies & Practical Application 4
A test publisher is developing a new workplace integrity test and wants to validate it.
Which validation strategy provides the strongest evidence for use in pre-employment screening?