ATP Quality Control & Assurance 5

Question 1

Which practice is a key component of quality control in automated essay scoring (AES) systems?

Accepted Answer

Comparing AES scores to trained human rater scores to establish concordance

Answer

Concordance studies comparing AES output to expert human rater judgments validate that the engine is measuring the intended construct before operational deployment.

Question 2

A testing organization implements a 'freeze period' for its item bank before each major administration. What quality purpose does this serve?

Accepted Answer

Preventing last-minute edits that could introduce errors into live forms

Answer

A freeze period locks item content and metadata so no unreviewed changes can propagate into an upcoming operational form, guarding against inadvertent errors close to administration.

Question 3

Which measure quantifies the precision of a single examinee's score and is used to construct score confidence intervals?

Accepted Answer

Standard error of measurement (SEM)

Answer

The SEM estimates how much an individual's observed score is expected to vary due to measurement error, allowing test users to express score precision as a confidence interval.

Question 4

In the context of ATP operational guidelines, 'test security incident response' requires which first step when a breach is detected?

Accepted Answer

Document the incident and notify designated security officers

Answer

Immediate documentation and notification of security officers initiates the formal incident response chain, enabling evidence preservation and an informed decision about remediation.

Question 5

A credentialing body notices that pass rates differ significantly across testing sites without a known examinee characteristic explanation. Which QA process should be triggered?

Accepted Answer

Administration irregularity investigation

Answer

Unexplained site-level pass rate discrepancies may indicate administration problems such as unauthorized assistance, scoring errors, or breaches requiring investigation.

ATP - Association of Test Publishers Practice Test

ATP - Association of Test Publishers Practice Test

ATP Quality Control & Assurance 5