ATP Quality Control & Assurance 3

Question 1

Which type of reliability coefficient is most appropriate for evaluating consistency of scores on a single administration of a multiple-choice test?

Accepted Answer

Internal consistency (e.g., coefficient alpha)

Answer

Internal consistency coefficients like Cronbach's alpha or KR-20 measure how consistently items measure the same construct within a single administration.

Question 2

A testing organization discovers a subset of items has been exposed on a social media site prior to the administration. Which quality control step should have prevented this?

Accepted Answer

Secure item banking with access controls

Answer

Robust access controls on the item bank, including role-based permissions and audit logs, are designed to prevent unauthorized disclosure of live items.

Question 3

Which validity evidence source involves examining whether test scores predict relevant future outcomes such as job performance?

Accepted Answer

Criterion-related (predictive) validity evidence

Answer

Predictive validity evidence demonstrates that scores are meaningfully related to a criterion measured at a later time, supporting score interpretations for selection or licensure.

Question 4

When conducting a standard-setting study, which practice supports quality assurance of panelist judgments?

Accepted Answer

Providing panelists with impact data and allowing discussion between rounds

Answer

Sharing impact data and facilitating structured discussion between rounds (as in Angoff or bookmark methods) helps panelists calibrate their judgments and reach defensible consensus.

Question 5

A QA checklist for item writing includes verifying that stems avoid 'window dressing.' What does this term mean?

Accepted Answer

Including irrelevant information that does not affect the answer

Answer

Window dressing refers to extraneous detail in the item stem that may confuse examinees or artificially inflate item difficulty without adding construct-relevant information.

ATP - Association of Test Publishers Practice Test

ATP - Association of Test Publishers Practice Test

ATP Quality Control & Assurance 3