CPE Evaluation Methods & Reporting 4 — Questions and Answers
Question 1: A program serves clients in urban, rural, and suburban settings. To ensure geographic representation in your evaluation sample, you should use:
- Simple random sampling
- Systematic sampling
- Stratified random sampling (Correct answer)
- Convenience sampling
Correct answer: Stratified random sampling
Stratified random sampling divides the population into subgroups (strata) and samples from each, ensuring all geographic settings are proportionally represented.
Question 2: Which of the following BEST describes a 'ripple effect' in program evaluation?
- Unintended negative consequences of a program intervention
- Indirect or secondary outcomes extending beyond intended direct beneficiaries (Correct answer)
- The spread of program implementation fidelity across sites
- Budget overruns that cascade through subsequent program years
Correct answer: Indirect or secondary outcomes extending beyond intended direct beneficiaries
Ripple effects are secondary, indirect outcomes that extend to people or systems beyond the program's direct participants, often captured in systems-level evaluations.
Question 3: An evaluator presents preliminary findings to program staff before finalizing the report. This practice is called:
- Member checking (Correct answer)
- Triangulation
- Peer debriefing
- Audit trail
Correct answer: Member checking
Member checking involves sharing preliminary findings with study participants or program stakeholders to verify accuracy and interpretation of data.
Question 4: When using Most Significant Change (MSC) technique, data are collected primarily through:
- Structured questionnaires with Likert scales
- Stories of change selected through deliberate stakeholder review (Correct answer)
- Administrative database extraction
- Observation protocols with standardized rating scales
Correct answer: Stories of change selected through deliberate stakeholder review
MSC collects and systematically selects stories about meaningful changes, making it a participatory qualitative monitoring and evaluation approach.
Question 5: The primary purpose of a logic model in program evaluation is to:
- Calculate the cost-benefit ratio of a program
- Depict the theory of how program inputs lead to intended outcomes (Correct answer)
- Identify statistical controls for confounding variables
- Determine the appropriate sample size for the evaluation
Correct answer: Depict the theory of how program inputs lead to intended outcomes
A logic model visually represents the program's theory of change, linking resources and activities to outputs, outcomes, and impact.
Question 6: In cost-effectiveness analysis, program outcomes are expressed as:
- Dollar values using willingness-to-pay methods
- Quality-adjusted life years (QALYs)
- Natural units (e.g., cost per dropout prevented) (Correct answer)
- Net present value of future earnings
Correct answer: Natural units (e.g., cost per dropout prevented)
Cost-effectiveness analysis keeps outcomes in their natural units (e.g., cost per unit of outcome), unlike cost-benefit analysis which monetizes all outcomes.
Question 7: A funder requires an evaluator to guarantee positive findings before contracting. The evaluator should:
- Accept if the program has strong preliminary evidence
- Decline and explain this violates evaluation ethics (Correct answer)
- Negotiate to report only statistically significant positive results
- Accept if the evaluation design is rigorous enough
Correct answer: Decline and explain this violates evaluation ethics
Guaranteeing findings in advance violates the AEA Guiding Principles for Evaluators, specifically integrity, honesty, and independence.
CPE Evaluation Methods & Reporting 4
A program serves clients in urban, rural, and suburban settings.
To ensure geographic representation in your evaluation sample, you should use: