Government Procurement Certification Program Evaluation 3 — Questions and Answers
Question 1: Which of the following is a key characteristic of a 'utilization-focused evaluation'?
- It prioritizes academic rigor over practical use
- It is designed from the start to ensure findings are actually used by decision-makers (Correct answer)
- It relies exclusively on quantitative data
- It is conducted only after a program ends
Correct answer: It is designed from the start to ensure findings are actually used by decision-makers
Utilization-focused evaluation, developed by Michael Quinn Patton, emphasizes designing evaluations so that findings are actively used by intended users for decision-making.
Question 2: In a procurement evaluation, 'cost-effectiveness analysis' differs from 'cost-benefit analysis' in that it:
- Measures only program costs, not benefits
- Compares costs to non-monetary outcomes rather than converting all values to dollars (Correct answer)
- Is used only for contracts over $10 million
- Requires a control group for validity
Correct answer: Compares costs to non-monetary outcomes rather than converting all values to dollars
Cost-effectiveness analysis compares costs to a single, often non-monetary outcome measure, while cost-benefit analysis converts all costs and benefits to monetary values.
Question 3: What does 'evaluation validity' refer to in the context of program assessment?
- Whether the evaluation was completed on time and within budget
- The degree to which the evaluation measures what it claims to measure (Correct answer)
- Whether all stakeholders approved the evaluation plan
- The legal authority to conduct the evaluation
Correct answer: The degree to which the evaluation measures what it claims to measure
Validity refers to the accuracy of the evaluation's measurements and conclusions — whether the tools and methods truly capture what they are intended to measure.
Question 4: A program evaluation finds statistically significant results but the effect size is very small. What should an evaluator conclude?
- The program is definitively successful and should be expanded
- The results are statistically significant but may not be practically meaningful (Correct answer)
- The evaluation methodology was flawed
- Small effect sizes always indicate program failure
Correct answer: The results are statistically significant but may not be practically meaningful
Statistical significance indicates results are unlikely due to chance, but effect size determines practical importance — small effects may not justify program costs.
Question 5: Which federal framework requires agencies to measure and report on program performance as part of the budget process?
- Federal Acquisition Regulation (FAR)
- Government Performance and Results Modernization Act (GPRAMA) (Correct answer)
- Chief Financial Officers Act
- Clinger-Cohen Act
Correct answer: Government Performance and Results Modernization Act (GPRAMA)
The Government Performance and Results Modernization Act (GPRAMA) of 2010 requires agencies to set strategic goals, measure performance, and report results to support budget decisions.
Question 6: What is the main purpose of a 'process evaluation' in government procurement programs?
- To determine the dollar value of program outcomes
- To examine whether the program is being implemented as designed (Correct answer)
- To compare the program against competing programs
- To evaluate the financial integrity of contractor payments
Correct answer: To examine whether the program is being implemented as designed
Process evaluation (also called implementation evaluation) examines whether program activities are being carried out as intended and identifies implementation gaps.
Question 7: When an evaluator uses 'triangulation' in a program evaluation, this means:
- Using geometric methods to analyze procurement data
- Cross-checking findings using multiple data sources or methods to increase confidence (Correct answer)
- Interviewing three stakeholders minimum for each finding
- Applying three separate budget forecasts
Correct answer: Cross-checking findings using multiple data sources or methods to increase confidence
Triangulation involves using multiple data sources, methods, or evaluators to verify findings and strengthen the credibility of conclusions.
Government Procurement Certification Program Evaluation 3
Which of the following is a key characteristic of a 'utilization-focused evaluation'?