KSA Assessment Design & Evaluation Methods 5 — Questions and Answers
Question 1: Under the Uniform Guidelines on Employee Selection Procedures (1978), adverse impact is generally indicated when the selection rate for a protected group is less than what fraction of the highest-scoring group?
- One-half (50%)
- Two-thirds (67%)
- Four-fifths (80%) (Correct answer)
- Nine-tenths (90%)
Correct answer: Four-fifths (80%)
The four-fifths (80%) rule in the Uniform Guidelines defines adverse impact as a selection rate for a protected group less than 80% of the rate for the group with the highest selection rate.
Question 2: A test with high specificity but low sensitivity in a selection context would:
- Accept most qualified candidates but also admit many unqualified ones
- Correctly reject unqualified candidates but miss many qualified ones (Correct answer)
- Balance false positives and false negatives equally
- Maximize the total number of candidates selected
Correct answer: Correctly reject unqualified candidates but miss many qualified ones
High specificity means the test rarely selects unqualified candidates (low false positives), but low sensitivity means it rejects many truly qualified candidates (high false negatives).
Question 3: The ADDIE model's 'Analysis' phase in assessment development primarily involves:
- Building test items and scoring rubrics
- Identifying performance gaps, learner characteristics, and required KSAs (Correct answer)
- Piloting the assessment with a representative sample
- Reviewing learner outcomes post-assessment
Correct answer: Identifying performance gaps, learner characteristics, and required KSAs
The Analysis phase defines the performance gap, target audience, and required KSAs that the assessment must address before any design work begins.
Question 4: Which type of rating scale requires raters to allocate a fixed percentage of employees into performance categories, preventing all employees from receiving the highest rating?
- Behaviorally Anchored Rating Scale (BARS)
- Graphic Rating Scale
- Forced distribution (Correct answer)
- Mixed Standard Scale
Correct answer: Forced distribution
Forced distribution mandates that ratings follow a predetermined curve (e.g., top 10%, middle 70%, bottom 20%), preventing rating inflation.
Question 5: A KSA assessor is reviewing a written knowledge test and notices that item stems are ambiguous and answer choices overlap. This most directly threatens:
- Criterion-related validity
- Test security
- Content and construct validity (Correct answer)
- Interrater reliability
Correct answer: Content and construct validity
Ambiguous stems and overlapping options introduce construct-irrelevant variance, undermining both content validity (unclear what is being measured) and construct validity (adding unintended factors like reading ambiguity).
Question 6: Which statistical technique is used to identify groups of test items that measure a common underlying factor?
- Multiple regression
- Factor analysis (Correct answer)
- Analysis of variance (ANOVA)
- Discriminant function analysis
Correct answer: Factor analysis
Factor analysis reveals the latent structure of a test by identifying clusters of items that share common variance, indicating they measure a common construct.
Question 7: The concept of 'assessment for learning' (formative) differs from 'assessment of learning' (summative) in that it is primarily designed to:
- Rank candidates for hiring decisions
- Provide ongoing feedback to improve performance during learning (Correct answer)
- Certify that a candidate meets minimum qualifications
- Generate normative data for future test development
Correct answer: Provide ongoing feedback to improve performance during learning
Formative assessment (assessment for learning) is embedded within the learning or development process to provide feedback that guides improvement, not to make final judgments.
KSA Assessment Design & Evaluation Methods 5
Under the Uniform Guidelines on Employee Selection Procedures (1978), adverse impact is generally indicated when the selection rate for a protected group is less than what fraction of the highest-scoring group?