MTS Evaluation 3 — Questions and Answers
Question 1: Which type of reliability is assessed by administering the same test to the same group on two separate occasions?
- Inter-rater reliability
- Internal consistency
- Test-retest reliability (Correct answer)
- Split-half reliability
Correct answer: Test-retest reliability
Test-retest reliability measures the consistency of results when the same instrument is administered twice to the same group.
Question 2: A training evaluation reveals that the skills gap has been eliminated but employee turnover has increased. At which Kirkpatrick level should the MTS investigate further?
- Level 1
- Level 2
- Level 3
- Level 4 (Correct answer)
Correct answer: Level 4
Level 4 (Results) examines organizational outcomes like turnover, productivity, and ROI, which may be affected by factors beyond training.
Question 3: What is the main advantage of using anonymous surveys in Level 1 reaction evaluations?
- They are easier to analyze statistically
- They encourage more honest and candid feedback (Correct answer)
- They eliminate social desirability bias in all cases
- They comply with federal training regulations
Correct answer: They encourage more honest and candid feedback
Anonymity reduces social desirability bias and encourages learners to provide candid, honest reactions to the training.
Question 4: Which approach involves evaluating training while it is still being developed and delivered for the first time?
- Summative evaluation
- Formative evaluation (Correct answer)
- Confirmative evaluation
- Goal-free evaluation
Correct answer: Formative evaluation
Formative evaluation is conducted during development or early delivery to identify and correct problems before full implementation.
Question 5: An evaluator uses a rubric to score role-play performance. What type of validity evidence does a well-designed rubric primarily support?
- Content validity (Correct answer)
- Predictive validity
- Concurrent validity
- Construct validity
Correct answer: Content validity
A rubric aligned with the job task content provides content validity evidence that the assessment covers the relevant domain.
Question 6: When calculating return on investment (ROI) for a training program, what does the Phillips ROI Methodology add beyond Kirkpatrick's four levels?
- A pre-training needs assessment step
- A fifth level that quantifies monetary impact (Correct answer)
- A peer review process for evaluation data
- A standardized survey instrument
Correct answer: A fifth level that quantifies monetary impact
Phillips extended Kirkpatrick's model with a Level 5 that converts Level 4 results to monetary value and calculates ROI percentage.
Question 7: Which data collection method is best suited for exploring complex reasons behind low training transfer rates?
- Multiple-choice post-test
- Structured observation checklist
- In-depth interview (Correct answer)
- Learning management system (LMS) completion report
Correct answer: In-depth interview
In-depth interviews allow evaluators to probe nuanced barriers to transfer that structured instruments cannot easily capture.
MTS Evaluation 3
Which type of reliability is assessed by administering the same test to the same group on two separate occasions?