ATP Quality Control & Assurance 5 — Questions and Answers
Question 1: Which practice is a key component of quality control in automated essay scoring (AES) systems?
- Relying solely on human raters for all final scores
- Comparing AES scores to trained human rater scores to establish concordance (Correct answer)
- Using only grammar-based features for scoring
- Administering AES only to high-stakes credentialing exams
Correct answer: Comparing AES scores to trained human rater scores to establish concordance
Concordance studies comparing AES output to expert human rater judgments validate that the engine is measuring the intended construct before operational deployment.
Question 2: A testing organization implements a 'freeze period' for its item bank before each major administration. What quality purpose does this serve?
- Allowing time to add new items
- Preventing last-minute edits that could introduce errors into live forms (Correct answer)
- Reducing licensing costs for the item bank software
- Enabling faster form assembly
Correct answer: Preventing last-minute edits that could introduce errors into live forms
A freeze period locks item content and metadata so no unreviewed changes can propagate into an upcoming operational form, guarding against inadvertent errors close to administration.
Question 3: Which measure quantifies the precision of a single examinee's score and is used to construct score confidence intervals?
- Standard deviation of item difficulty
- Standard error of measurement (SEM) (Correct answer)
- Coefficient of variation
- Cronbach's alpha
Correct answer: Standard error of measurement (SEM)
The SEM estimates how much an individual's observed score is expected to vary due to measurement error, allowing test users to express score precision as a confidence interval.
Question 4: In the context of ATP operational guidelines, 'test security incident response' requires which first step when a breach is detected?
- Publicly announce the breach immediately
- Document the incident and notify designated security officers (Correct answer)
- Cancel all scores from the affected administration
- Replace all items in the item bank
Correct answer: Document the incident and notify designated security officers
Immediate documentation and notification of security officers initiates the formal incident response chain, enabling evidence preservation and an informed decision about remediation.
Question 5: A credentialing body notices that pass rates differ significantly across testing sites without a known examinee characteristic explanation. Which QA process should be triggered?
- Changing the cut score for affected sites
- Administration irregularity investigation (Correct answer)
- Revising item difficulty targets
- Switching to a paper-based format
Correct answer: Administration irregularity investigation
Unexplained site-level pass rate discrepancies may indicate administration problems such as unauthorized assistance, scoring errors, or breaches requiring investigation.
Question 6: According to ATP best practices, which stakeholder group should be involved in validating a job task analysis (JTA) used to build a certification exam blueprint?
- Only psychometricians and test developers
- Subject matter experts (SMEs) actively working in the profession (Correct answer)
- Exam candidates who have recently passed the exam
- Third-party accreditation reviewers only
Correct answer: Subject matter experts (SMEs) actively working in the profession
SMEs currently practicing in the field provide the occupational knowledge needed to confirm that the JTA accurately reflects critical tasks and the knowledge required to perform them.
Question 7: Which equating design uses a set of common items appearing on both old and new forms to link their score scales?
- Single-group equating
- Counterbalanced equating
- Common-item nonequivalent groups (anchor test) design (Correct answer)
- Random-groups equating
Correct answer: Common-item nonequivalent groups (anchor test) design
The anchor test design embeds a subset of previously calibrated items in each new form; these items provide the statistical bridge needed to place new-form scores on the established scale.
ATP Quality Control & Assurance 5
Which practice is a key component of quality control in automated essay scoring (AES) systems?