ATP Case Studies & Practical Application 5 — Questions and Answers
Question 1: A testing organization is evaluating whether to adopt a new computerized adaptive testing (CAT) platform. Which factor is most critical to evaluate from a measurement validity perspective?
- The cost savings compared to linear testing
- Whether the item response theory model underlying the CAT accurately fits the item bank and whether adaptive algorithms produce reliable scores across the ability range (Correct answer)
- The aesthetic design of the user interface
- Whether the platform vendor is ISO-certified
Correct answer: Whether the item response theory model underlying the CAT accurately fits the item bank and whether adaptive algorithms produce reliable scores across the ability range
CAT validity depends on IRT model-data fit and the adaptive algorithm's ability to produce reliable, unbiased ability estimates across the full score range.
Question 2: An organization finds that its high-stakes exam consistently produces higher scores for examinees who take the test multiple times, even on alternate forms. What is the primary validity threat this represents?
- Practice effects and construct-irrelevant familiarity with test format inflating scores beyond true ability gains (Correct answer)
- The alternate forms are not parallel
- Examinees are sharing specific item content between attempts
- The scoring rubric is too lenient on retakes
Correct answer: Practice effects and construct-irrelevant familiarity with test format inflating scores beyond true ability gains
Systematic score gains on retakes beyond true learning represent practice effects, a form of construct-irrelevant variance that can inflate scores and compromise score meaning.
Question 3: A test publisher is launching an exam in a new country where the target profession is regulated differently than in the US. Which step is most important before adapting the existing exam?
- Translate the exam using the existing US item bank
- Conduct a job task analysis in the new country to determine whether the competency domain and standards are sufficiently similar to support score-based inferences (Correct answer)
- Use the US passing standard directly since it is already validated
- Survey US-certified professionals for their opinion on the new country's standards
Correct answer: Conduct a job task analysis in the new country to determine whether the competency domain and standards are sufficiently similar to support score-based inferences
A local job task analysis is required to establish whether the construct being measured is the same in the new regulatory context before adapting or deploying the exam.
Question 4: A testing organization's internal audit reveals that its score reporting turnaround time is 6 weeks, causing candidates to delay career decisions. How should the organization approach improving this without compromising score validity?
- Release preliminary scores immediately and correct them later if needed
- Analyze the scoring pipeline to identify bottlenecks while maintaining all required psychometric quality control checkpoints before official score release (Correct answer)
- Eliminate equating to speed up score reporting
- Report only pass/fail decisions quickly and withhold scaled scores
Correct answer: Analyze the scoring pipeline to identify bottlenecks while maintaining all required psychometric quality control checkpoints before official score release
Process analysis to reduce operational inefficiencies is appropriate, but no psychometric quality control step—such as equating, DIF review, or score verification—should be eliminated in the interest of speed.
Question 5: A certification board is considering using social media activity as a supplemental data source in its exam fraud investigation process. What is the primary ethical and legal concern?
- Social media data is too expensive to collect at scale
- Using social media data raises significant privacy, due process, and fairness concerns and must be governed by clearly defined, disclosed policies and legal review (Correct answer)
- Social media cannot provide useful information about test fraud
- Examinees would object publicly, creating reputational risk
Correct answer: Using social media data raises significant privacy, due process, and fairness concerns and must be governed by clearly defined, disclosed policies and legal review
Social media monitoring in credentialing contexts raises due process, privacy, and disparate impact concerns that require formal policy frameworks and legal vetting before use.
Question 6: A testing organization administers the same exam in both English and Mandarin Chinese. A DIF analysis reveals that several items function differently across language groups. What is the most appropriate action?
- Remove all Mandarin-language examinees from the scoring sample
- Investigate the flagged items for translation artifacts, cultural bias, or construct-irrelevant difficulty, and revise or remove items that show unjustifiable DIF (Correct answer)
- Ignore DIF because bilingual exams always show some differences
- Report separate scaled scores for each language version without further investigation
Correct answer: Investigate the flagged items for translation artifacts, cultural bias, or construct-irrelevant difficulty, and revise or remove items that show unjustifiable DIF
DIF flags require content review to distinguish between items that are legitimately harder in one language versus items with translation errors or cultural bias requiring revision or removal.
Question 7: A state licensure board is under political pressure to reduce barriers to entry by lowering the exam passing standard. What is the board's most defensible response?
- Comply with the political pressure to maintain good relations with the legislature
- Explain that the passing standard is set to protect public safety based on empirical job analysis and standard-setting evidence, and that any change must follow the same systematic process (Correct answer)
- Conduct a quick survey of current licensees to see if they support a lower standard
- Lower the standard temporarily and re-evaluate in one year
Correct answer: Explain that the passing standard is set to protect public safety based on empirical job analysis and standard-setting evidence, and that any change must follow the same systematic process
Credentialing bodies must defend empirically established standards by articulating the public safety rationale and the systematic process used, while acknowledging that any revision must follow equally rigorous methodology.
ATP Case Studies & Practical Application 5
A testing organization is evaluating whether to adopt a new computerized adaptive testing (CAT) platform.
Which factor is most critical to evaluate from a measurement validity perspective?