CPE Evaluation Methods & Reporting 5 — Questions and Answers
Question 1: Which design provides the strongest evidence of causal attribution in program evaluation?
- Pre-post single group design
- Matched comparison group design
- Randomized controlled trial (Correct answer)
- Interrupted time series without control group
Correct answer: Randomized controlled trial
Randomized controlled trials (RCTs) use random assignment to create equivalent groups, providing the strongest control for selection bias and confounding variables.
Question 2: Regression discontinuity design (RDD) is appropriate when:
- Program participants are assigned based on a continuous cutoff score (Correct answer)
- Two groups are perfectly matched on all demographic variables
- The program uses a lottery to determine participant selection
- Outcomes are measured at three or more time points
Correct answer: Program participants are assigned based on a continuous cutoff score
RDD exploits a cutoff score (e.g., income threshold, test score) used for program assignment to estimate causal effects at the threshold.
Question 3: Social Return on Investment (SROI) differs from traditional cost-benefit analysis primarily because SROI:
- Uses only government-approved economic data sources
- Monetizes social, environmental, and economic value using stakeholder-defined proxies (Correct answer)
- Excludes unintended consequences from calculations
- Requires a randomized control group for validity
Correct answer: Monetizes social, environmental, and economic value using stakeholder-defined proxies
SROI assigns monetary values to social and environmental outcomes using stakeholder-defined proxies, broadening the scope beyond traditional economic metrics.
Question 4: An evaluator conducting focus groups notices one dominant participant repeatedly silencing others. The BEST immediate response is to:
- End the focus group and replace it with individual interviews
- Use structured turn-taking techniques to ensure all voices are heard (Correct answer)
- Remove the dominant participant from the group
- Note the dynamic in field notes and continue without intervention
Correct answer: Use structured turn-taking techniques to ensure all voices are heard
Using facilitation techniques like structured turn-taking or direct invitation addresses dominance dynamics while preserving the group discussion.
Question 5: The AEA Program Evaluation Standards include which of the following categories?
- Validity, reliability, generalizability, and significance
- Utility, feasibility, propriety, accuracy, and evaluation accountability (Correct answer)
- Formative, summative, developmental, and empowerment standards
- Quantitative, qualitative, mixed-methods, and economic standards
Correct answer: Utility, feasibility, propriety, accuracy, and evaluation accountability
The Joint Committee's Program Evaluation Standards include five categories: utility, feasibility, propriety, accuracy, and evaluation accountability.
Question 6: When an evaluator discovers mid-study that the program significantly changed its intervention model, the MOST appropriate action is to:
- Continue the original evaluation plan to maintain research integrity
- Terminate the evaluation and return all funding
- Document the change, revise the evaluation questions, and renegotiate the evaluation plan with stakeholders (Correct answer)
- Analyze only data collected before the program change
Correct answer: Document the change, revise the evaluation questions, and renegotiate the evaluation plan with stakeholders
Mid-program changes require the evaluator to adapt the evaluation plan and renegotiate with stakeholders to ensure the evaluation remains relevant and valid.
Question 7: Which technique is used to synthesize findings across multiple evaluations of similar programs to draw broader conclusions?
- Case study analysis
- Meta-analysis or systematic review (Correct answer)
- Cross-site evaluation
- Realist synthesis
Correct answer: Meta-analysis or systematic review
Meta-analysis statistically pools effect sizes across multiple studies, and systematic reviews synthesize evidence, both producing generalizable conclusions about program effectiveness.
CPE Evaluation Methods & Reporting 5
Which design provides the strongest evidence of causal attribution in program evaluation?