KSA Assessment Design & Evaluation Methods 4 — Questions and Answers
Question 1: Which scale of measurement allows for meaningful calculation of ratios between scores?
- Nominal
- Ordinal
- Interval
- Ratio (Correct answer)
Correct answer: Ratio
Ratio scales have a true zero point, making it meaningful to say one score is twice another, unlike interval scales which lack a true zero.
Question 2: A KSA assessor notices an assessment consistently produces scores 10 points higher than a validated benchmark tool. This is an example of:
- Random error
- Systematic error (bias) (Correct answer)
- Low internal consistency
- Poor inter-rater reliability
Correct answer: Systematic error (bias)
Systematic error (bias) causes scores to consistently deviate in one direction from the true score, unlike random error which fluctuates unpredictably.
Question 3: The three-parameter logistic (3PL) model in Item Response Theory adds which parameter compared to the 1PL (Rasch) model?
- Item discrimination
- Guessing (pseudo-chance level) (Correct answer)
- Test reliability
- Sample size adjustment
Correct answer: Guessing (pseudo-chance level)
The 3PL model extends the 1PL by adding a discrimination parameter and a pseudo-guessing parameter (c) that accounts for the probability of low-ability examinees answering correctly by chance.
Question 4: A work sample test requires candidates to perform tasks that directly replicate key job duties. Its primary advantage is:
- Low development cost
- High fidelity and content validity (Correct answer)
- Ease of standardized scoring
- Minimal time requirements
Correct answer: High fidelity and content validity
Work sample tests have high content validity because they closely mirror actual job tasks, increasing both predictive validity and candidate acceptance.
Question 5: Which type of validation study compares test scores to performance measures collected at the same time from current employees?
- Predictive validity study
- Concurrent validity study (Correct answer)
- Synthetic validity study
- Construct validity study
Correct answer: Concurrent validity study
A concurrent validity study collects both predictor and criterion data simultaneously from a current workforce, making it faster but more vulnerable to range restriction.
Question 6: When an organization cannot conduct a local validity study due to small sample size, which approach is most appropriate?
- Skip validation entirely
- Conduct a transportability study using validity generalization evidence (Correct answer)
- Use only unstructured interviews
- Reduce the number of predictors to one
Correct answer: Conduct a transportability study using validity generalization evidence
Validity generalization allows organizations to use meta-analytic evidence showing that a predictor's validity generalizes across settings when local studies are infeasible.
Question 7: In assessment center design, which exercise best evaluates an assessor's ability to handle competing priorities and manage time under pressure?
- Leaderless group discussion
- In-basket (in-tray) exercise (Correct answer)
- Role-play simulation
- Presentation exercise
Correct answer: In-basket (in-tray) exercise
The in-basket exercise presents a series of memos, emails, and tasks that must be prioritized and acted upon within a time limit, directly simulating workload management.
KSA Assessment Design & Evaluation Methods 4
Which scale of measurement allows for meaningful calculation of ratios between scores?