CAE Evaluation Strategies 3 — Questions and Answers
Question 1: An adult education program director wants to determine whether the program is achieving its mission over multiple years. The MOST appropriate evaluation approach is:
- Single-session formative feedback
- Longitudinal outcome evaluation (Correct answer)
- Immediate reaction surveys
- Instructor self-assessment
Correct answer: Longitudinal outcome evaluation
Longitudinal outcome evaluation tracks participant outcomes over time, making it suitable for assessing long-term program mission achievement.
Question 2: Which evaluation strategy is MOST aligned with an empowerment evaluation philosophy?
- External evaluators control all data collection and reporting
- Program stakeholders actively participate in designing and conducting the evaluation (Correct answer)
- Evaluation results are used solely for funding decisions
- Only quantitative data is collected to ensure objectivity
Correct answer: Program stakeholders actively participate in designing and conducting the evaluation
Empowerment evaluation involves stakeholders in the process to build their capacity for self-assessment and program improvement.
Question 3: A needs assessment reveals a gap between current and desired performance. In evaluation terms, this gap represents:
- A learning objective
- The basis for selecting evaluation instruments
- The discrepancy that justifies the program's existence (Correct answer)
- An outcome indicator
Correct answer: The discrepancy that justifies the program's existence
A performance gap identified in needs assessment documents the discrepancy that the educational program is designed to address, which also frames the evaluation questions.
Question 4: Portfolio assessment is particularly suited to adult learners because it:
- Allows rapid standardized scoring
- Captures growth over time and values prior experience (Correct answer)
- Eliminates subjectivity in grading
- Focuses exclusively on knowledge recall
Correct answer: Captures growth over time and values prior experience
Portfolios document learning progression and can incorporate prior knowledge artifacts, honoring the experiential backgrounds adult learners bring.
Question 5: When an evaluation finds that a program worked well for one subgroup but not another, this is referred to as:
- A Type I error
- A moderator effect (Correct answer)
- Response bias
- Construct invalidity
Correct answer: A moderator effect
A moderator effect occurs when the relationship between a program and its outcomes differs across subgroups such as age, gender, or prior education.
Question 6: A CAE practitioner uses think-aloud protocols while adult learners complete a task. This qualitative evaluation technique PRIMARILY captures:
- Final performance scores
- Cognitive processes and reasoning during task completion (Correct answer)
- Peer comparison data
- Instructor delivery quality
Correct answer: Cognitive processes and reasoning during task completion
Think-aloud protocols elicit verbal descriptions of thought processes, revealing how learners approach problems rather than just whether they succeed.
Question 7: Which threat to internal validity occurs when historical events outside the program influence participant outcomes?
- Maturation
- Instrumentation
- History (Correct answer)
- Selection bias
Correct answer: History
The history threat occurs when external events—such as a policy change or economic shift—coincide with the program and confound measured outcomes.
CAE Evaluation Strategies 3
An adult education program director wants to determine whether the program is achieving its mission over multiple years.
The MOST appropriate evaluation approach is: