CSP Evaluation Research & Impact Assessment 3 — Questions and Answers
Question 1: In regression discontinuity design, program impact is estimated by:
- Comparing all participants to all non-participants
- Examining outcome differences at the cutoff point of an assignment variable (Correct answer)
- Randomly assigning participants to treatment and control groups
- Matching participants on propensity scores
Correct answer: Examining outcome differences at the cutoff point of an assignment variable
Regression discontinuity exploits the arbitrary cutoff of an eligibility criterion to estimate causal effects.
Question 2: Stake's responsive evaluation model prioritizes:
- Pre-specified behavioral objectives and measurable goals
- Stakeholder concerns and program issues as they emerge during evaluation (Correct answer)
- Experimental control and random assignment
- Economic efficiency and cost savings
Correct answer: Stakeholder concerns and program issues as they emerge during evaluation
Responsive evaluation (Stake) adapts to the concerns and issues that stakeholders raise throughout the evaluation process.
Question 3: Social Return on Investment (SROI) differs from traditional cost-benefit analysis primarily by:
- Focusing exclusively on government program savings
- Assigning monetary values to social and environmental outcomes (Correct answer)
- Excluding beneficiary perspectives from the analysis
- Measuring only short-term economic impacts
Correct answer: Assigning monetary values to social and environmental outcomes
SROI monetizes broader social and environmental value, including outcomes for communities and society.
Question 4: In most-similar systems design (comparative evaluation), researchers:
- Select cases that differ on most variables to find universal patterns
- Choose cases that are similar on key background factors but differ on outcomes (Correct answer)
- Randomly assign policies to geographic units for comparison
- Focus on a single case to maximize contextual depth
Correct answer: Choose cases that are similar on key background factors but differ on outcomes
Most-similar design controls for background variables by choosing comparable contexts, then explains why outcomes differ.
Question 5: An evaluator discovers a program is producing significant unintended positive effects not in its logic model. The most appropriate action is to:
- Ignore them since they fall outside the evaluation scope
- Document and report them as emergent outcomes alongside intended outcomes (Correct answer)
- Exclude them to avoid scope creep in the final report
- Only report them if the program director requests it
Correct answer: Document and report them as emergent outcomes alongside intended outcomes
Professional evaluation standards require reporting significant unintended outcomes, both positive and negative.
Question 6: The primary purpose of a formative evaluation is to:
- Determine whether a program should be continued or terminated
- Provide ongoing feedback to improve the program during implementation (Correct answer)
- Assess the long-term sustainability of program outcomes
- Satisfy accountability requirements for funders
Correct answer: Provide ongoing feedback to improve the program during implementation
Formative evaluation generates real-time information to guide program improvement while it is still running.
Question 7: Instrumental use of evaluation findings refers to:
- Direct application of findings to make specific program decisions (Correct answer)
- Broader shifts in stakeholder thinking and understanding
- Political use of findings to justify pre-existing decisions
- Publication of results in peer-reviewed journals
Correct answer: Direct application of findings to make specific program decisions
Instrumental use occurs when evaluation findings directly inform concrete decisions, such as program modifications or budget allocations.
CSP Evaluation Research & Impact Assessment 3
In regression discontinuity design, program impact is estimated by: