CPE Evaluation Methods & Reporting 3 — Questions and Answers
Question 1: An evaluator calculates a Cronbach's alpha of 0.58 for a program satisfaction survey. This result indicates:
- Excellent internal consistency requiring no changes
- Questionable reliability suggesting items should be revised (Correct answer)
- Strong test-retest reliability over time
- High construct validity of the instrument
Correct answer: Questionable reliability suggesting items should be revised
Cronbach's alpha below 0.70 is generally considered questionable reliability, indicating the survey items may not consistently measure the same construct.
Question 2: Dose-response analysis in program evaluation examines:
- Side effects of intervention components on participants
- The relationship between the amount of program exposure and outcome magnitude (Correct answer)
- Pharmaceutical safety protocols adapted for social programs
- Optimal staff-to-participant ratios for program fidelity
Correct answer: The relationship between the amount of program exposure and outcome magnitude
Dose-response analysis explores whether greater program participation (dose) is associated with stronger outcomes (response), supporting causal inference.
Question 3: Which evaluation approach is BEST suited for assessing a mature, well-defined program with stable implementation across multiple sites?
- Developmental evaluation
- Formative evaluation
- Summative outcome evaluation (Correct answer)
- Needs assessment
Correct answer: Summative outcome evaluation
Summative outcome evaluation is appropriate for mature, stable programs where the goal is to judge overall effectiveness and generalizability of results.
Question 4: The concept of 'evaluation capacity building' refers to:
- Increasing the budget allocated for evaluation activities
- Developing organizations' skills and systems to conduct ongoing self-evaluation (Correct answer)
- Hiring additional external evaluators for larger studies
- Expanding the physical infrastructure for data storage
Correct answer: Developing organizations' skills and systems to conduct ongoing self-evaluation
Evaluation capacity building aims to strengthen organizations' internal ability to conduct, use, and champion evaluation as a regular practice.
Question 5: A researcher finds a statistically significant difference between groups (p=0.03) but Cohen's d is 0.08. The evaluator should conclude that:
- The intervention has a meaningful practical impact
- The finding is statistically significant but has negligible practical effect size (Correct answer)
- The study lacks sufficient statistical power
- The results are not trustworthy due to measurement error
Correct answer: The finding is statistically significant but has negligible practical effect size
A Cohen's d of 0.08 represents a trivial effect size; statistical significance with large samples does not guarantee practical or program relevance.
Question 6: In participatory evaluation, stakeholders are involved in which phases?
- Only data collection to prevent bias in design
- Only reporting to ensure findings are understood
- Multiple phases including design, data collection, and interpretation (Correct answer)
- Only the initial needs assessment phase
Correct answer: Multiple phases including design, data collection, and interpretation
Participatory evaluation engages stakeholders throughout the entire evaluation process to increase relevance, ownership, and utilization of findings.
Question 7: Which technique helps evaluators identify the most critical program components by systematically eliminating one element at a time to observe its effect?
- Contribution analysis
- Component analysis (dismantling design) (Correct answer)
- Regression discontinuity design
- Interrupted time series
Correct answer: Component analysis (dismantling design)
Component analysis (or dismantling design) isolates active ingredients by sequentially removing program elements to determine each component's contribution to outcomes.
CPE Evaluation Methods & Reporting 3
An evaluator calculates a Cronbach's alpha of 0.58 for a program satisfaction survey.
This result indicates: