MTS Assessment & Evaluation Design 2 — Questions and Answers
Question 1: A training manager wants to determine whether a new leadership course produced measurable behavior change on the job. Which Kirkpatrick level should be the primary focus?
- Level 1 – Reaction
- Level 2 – Learning
- Level 3 – Behavior (Correct answer)
- Level 4 – Results
Correct answer: Level 3 – Behavior
Level 3 (Behavior) measures the transfer of learned skills to the workplace after training.
Question 2: Which type of validity evidence is gathered by comparing test scores with scores on a well-established instrument measuring the same construct?
- Content validity
- Concurrent validity (Correct answer)
- Face validity
- Predictive validity
Correct answer: Concurrent validity
Concurrent validity is established by correlating new test scores with scores from an existing valid measure administered at the same time.
Question 3: A needs assessment reveals a performance gap caused by lack of motivation rather than skill deficiency. What is the most appropriate recommendation?
- Design a technical skills course
- Implement a training program with job aids
- Recommend a non-training intervention such as incentives (Correct answer)
- Increase the frequency of assessments
Correct answer: Recommend a non-training intervention such as incentives
When a gap is motivational rather than skill-based, non-training interventions like incentives or consequence management are more appropriate.
Question 4: In item analysis, a discrimination index of -0.15 for a test question indicates:
- The item effectively separates high and low performers
- High scorers are answering the item incorrectly more than low scorers (Correct answer)
- The item has an acceptable difficulty level
- The item should be retained without revision
Correct answer: High scorers are answering the item incorrectly more than low scorers
A negative discrimination index means higher-performing examinees are getting the item wrong more often than lower-performing examinees, suggesting a flawed item.
Question 5: Which evaluation design uses a pretest, training, posttest sequence with a separate untrained control group to isolate training effects?
- One-shot case study
- Time-series design
- Pre-experimental design
- True experimental design with control group (Correct answer)
Correct answer: True experimental design with control group
A true experimental design with random assignment and a control group provides the strongest evidence that training caused the observed change.
Question 6: Which scoring approach uses clearly described performance levels (e.g., Novice, Proficient, Expert) to rate observable skills?
- Norm-referenced scoring
- Criterion-referenced rubric (Correct answer)
- Holistic impression scoring
- Percentile ranking
Correct answer: Criterion-referenced rubric
A criterion-referenced rubric defines specific performance levels against fixed standards rather than comparing learners to each other.
Question 7: A portfolio assessment is best suited for measuring:
- Recall of isolated facts
- Speed of task completion
- Growth and cumulative demonstration of competency over time (Correct answer)
- Standardized knowledge across a large population
Correct answer: Growth and cumulative demonstration of competency over time
Portfolios document multiple work samples across time, making them ideal for assessing development and integrated competency.
MTS Assessment & Evaluation Design 2
A training manager wants to determine whether a new leadership course produced measurable behavior change on the job.
Which Kirkpatrick level should be the primary focus?