CPE Data Collection & Analysis 3 — Questions and Answers
Question 1: A program evaluator needs to assess behavioral changes in youth after an intervention. Which data source provides the most direct evidence?
- Staff satisfaction surveys
- Direct observation of participant behavior (Correct answer)
- Grant funding records
- Marketing materials
Correct answer: Direct observation of participant behavior
Direct observation provides firsthand evidence of actual behavioral changes rather than self-reported or proxy data.
Question 2: What does 'unit of analysis' refer to in a program evaluation?
- The statistical test used to analyze data
- The entity (individual, group, organization) about which data are collected (Correct answer)
- The timeframe for data collection
- The budget allocated for analysis
Correct answer: The entity (individual, group, organization) about which data are collected
The unit of analysis is the entity whose characteristics are being examined, such as individuals, groups, or programs.
Question 3: An evaluator notices that participants score higher on a measure simply because they've taken the same test before. This threat to validity is called:
- Selection bias
- Instrumentation error
- Testing effect (Correct answer)
- Regression to the mean
Correct answer: Testing effect
The testing effect occurs when prior exposure to a measure artificially inflates subsequent scores.
Question 4: Which sampling strategy ensures every member of the population has an equal chance of being selected?
- Convenience sampling
- Snowball sampling
- Simple random sampling (Correct answer)
- Purposive sampling
Correct answer: Simple random sampling
Simple random sampling gives every population member an equal probability of selection, reducing selection bias.
Question 5: In content analysis of qualitative data, 'coding' refers to:
- Encrypting sensitive data for privacy
- Assigning labels to segments of data to identify themes (Correct answer)
- Entering data into a spreadsheet
- Programming the evaluation database
Correct answer: Assigning labels to segments of data to identify themes
Coding in qualitative analysis involves systematically labeling data segments to identify patterns and themes.
Question 6: A CPE evaluator discovers that data collectors recorded information inconsistently across sites. This is an example of which threat to data quality?
- Attrition
- Instrumentation (Correct answer)
- Maturation
- History
Correct answer: Instrumentation
Instrumentation threat occurs when changes in measurement procedures or observers affect data consistency and validity.
Question 7: Which approach to data analysis is most appropriate when the evaluator wants to generate hypotheses rather than test them?
- Confirmatory factor analysis
- Hypothesis testing with null hypotheses
- Exploratory data analysis (Correct answer)
- Structural equation modeling
Correct answer: Exploratory data analysis
Exploratory data analysis examines data patterns without prespecified hypotheses, making it ideal for hypothesis generation.
CPE Data Collection & Analysis 3
A program evaluator needs to assess behavioral changes in youth after an intervention.
Which data source provides the most direct evidence?