KSA Assessment Design & Evaluation Methods 2 — Questions and Answers
Question 1: Which evaluation model emphasizes the use of goals-free evaluation to avoid bias toward intended outcomes?
- Kirkpatrick Model
- Scriven's Goal-Free Evaluation (Correct answer)
- CIPP Model
- Logic Model
Correct answer: Scriven's Goal-Free Evaluation
Scriven's Goal-Free Evaluation assesses actual effects without being anchored to the program's stated goals, reducing evaluator bias.
Question 2: A KSA assessor needs to determine if a selection test is measuring the intended construct. Which validity type is most relevant?
- Criterion-related validity
- Face validity
- Construct validity (Correct answer)
- Content validity
Correct answer: Construct validity
Construct validity confirms that an assessment actually measures the underlying theoretical construct it is designed to measure.
Question 3: In criterion-referenced assessment, a passing score is set by:
- The average score of the test-taking group
- A predetermined standard linked to job performance (Correct answer)
- Rank ordering test-takers
- The median score of a norm group
Correct answer: A predetermined standard linked to job performance
Criterion-referenced assessments use a fixed performance standard tied to job requirements rather than comparing candidates to each other.
Question 4: Which item format is most susceptible to test-wiseness, where examinees can identify the correct answer without content knowledge?
- Short answer
- Essay
- Multiple choice (Correct answer)
- Structured interview
Correct answer: Multiple choice
Multiple-choice items are vulnerable to test-wiseness strategies like eliminating obviously wrong distractors or recognizing grammatical cues.
Question 5: A Delphi method is used in assessment development primarily to:
- Analyze test score distributions
- Reach expert consensus on KSA definitions or cut scores (Correct answer)
- Compare norm groups across regions
- Calculate inter-rater reliability
Correct answer: Reach expert consensus on KSA definitions or cut scores
The Delphi method uses structured rounds of expert input and feedback to build consensus on complex judgments like job-relevant competencies or passing standards.
Question 6: An assessment that produces the same results regardless of who administers it demonstrates high:
- Internal consistency
- Scorer reliability (Correct answer)
- Test-retest reliability
- Parallel-form reliability
Correct answer: Scorer reliability
Scorer (or inter-rater) reliability refers to consistency of results across different administrators or raters applying the same assessment.
Question 7: Which step in the test development process involves verifying that items collectively represent all critical job tasks?
- Item banking
- Content validation (Correct answer)
- Norm referencing
- Adverse impact analysis
Correct answer: Content validation
Content validation ensures that test items adequately and proportionally cover all relevant KSAs and job tasks identified in a job analysis.
KSA Assessment Design & Evaluation Methods 2
Which evaluation model emphasizes the use of goals-free evaluation to avoid bias toward intended outcomes?