CT Evaluation Methods 5 — Questions and Answers
Question 1: Which evaluation strategy is MOST effective for assessing higher-order thinking skills such as analysis and application?
- True/false quiz
- Scenario-based case study assessment (Correct answer)
- Attendance tracking
- Factual recall multiple-choice test
Correct answer: Scenario-based case study assessment
Scenario-based assessments require learners to analyze situations and apply concepts, testing higher Bloom's taxonomy levels beyond recall.
Question 2: A trainer notices that post-test scores improved significantly but supervisor ratings of employee performance did not change. The MOST likely explanation is:
- The post-test was too easy
- There is a transfer problem — knowledge gained did not translate to behavior change (Correct answer)
- The Kirkpatrick Level 4 data was collected too soon
- Supervisor ratings are not a valid data source
Correct answer: There is a transfer problem — knowledge gained did not translate to behavior change
A gap between Level 2 (Learning) and Level 3 (Behavior) results indicates that conditions supporting transfer of training are lacking.
Question 3: When designing evaluation for a soft-skills training program (e.g., communication), a trainer should expect that measuring outcomes will be:
- Easy because communication is universally defined
- More challenging due to the subjective and behavioral nature of the skills (Correct answer)
- Unnecessary if participants reported high satisfaction
- Best handled by a single multiple-choice exam
Correct answer: More challenging due to the subjective and behavioral nature of the skills
Soft skills involve nuanced behaviors that are difficult to quantify, requiring multiple methods such as behavioral observation and 360-degree feedback.
Question 4: Which stakeholder is MOST important to involve when defining Level 4 (Results) evaluation metrics for a training program?
- The instructional designer
- Business leaders and sponsors who own the organizational outcomes (Correct answer)
- The IT department managing the LMS
- The learners who will attend the training
Correct answer: Business leaders and sponsors who own the organizational outcomes
Business leaders define what organizational success looks like, making them essential partners in identifying meaningful Level 4 outcome metrics.
Question 5: What is 'inter-rater reliability' in the context of performance-based training evaluation?
- The consistency of scores when different assessors evaluate the same learner performance (Correct answer)
- The degree to which a test measures what it is intended to measure
- The percentage of learners who agree with their own scores
- The correlation between pre-test and post-test results
Correct answer: The consistency of scores when different assessors evaluate the same learner performance
Inter-rater reliability ensures that multiple evaluators observing the same performance assign consistent scores, reducing subjectivity.
Question 6: A trainer is asked to evaluate a training program that was delivered six months ago with no baseline data collected. The BEST approach is to:
- Declare the evaluation impossible and stop
- Use retrospective pre-test/post-test where participants estimate prior knowledge (Correct answer)
- Administer a new post-test and assume it reflects change
- Only report Level 1 satisfaction data from post-training surveys
Correct answer: Use retrospective pre-test/post-test where participants estimate prior knowledge
A retrospective pre-test asks participants to estimate their knowledge level before training, allowing a comparison when baseline data was not originally collected.
Question 7: Which principle should guide how many evaluation methods a trainer selects for a single training program?
- Use only one method to keep evaluation simple
- Use every available method regardless of cost or purpose
- Select methods based on evaluation questions, resources, and what decisions the data must support (Correct answer)
- Always default to the cheapest available method
Correct answer: Select methods based on evaluation questions, resources, and what decisions the data must support
Evaluation method selection should be driven by the specific questions that need answering, practical constraints, and how the data will inform decisions.
CT Evaluation Methods 5
Which evaluation strategy is MOST effective for assessing higher-order thinking skills such as analysis and application?