PAC Research Methodologies & Data Analysis 3 ā Questions and Answers
Question 1: A policy analyst is studying the effect of a minimum wage increase using data from states that adopted the policy and those that did not. This is an example of:
- A randomized experiment
- A quasi-experimental design (Correct answer)
- A purely descriptive study
- A case-control study
Correct answer: A quasi-experimental design
Quasi-experimental designs exploit natural variation in policy adoption (rather than random assignment) to estimate causal effects.
Question 2: In content analysis of policy documents, what is intercoder reliability used to assess?
- Whether the documents are internally consistent
- The degree to which different coders assign the same codes to the same content (Correct answer)
- The statistical significance of themes identified
- The length and complexity of the documents analyzed
Correct answer: The degree to which different coders assign the same codes to the same content
Intercoder reliability (e.g., Cohen's kappa) measures agreement between multiple coders, ensuring the coding scheme is applied consistently.
Question 3: Which statistical test is most appropriate for comparing the mean outcomes of three or more independent policy programs?
- Paired t-test
- Chi-square test
- One-way ANOVA (Correct answer)
- Pearson correlation
Correct answer: One-way ANOVA
One-way ANOVA tests whether the means of three or more independent groups differ significantly, making it ideal for comparing multiple programs.
Question 4: A regression model predicting recidivism rates has an RĀ² of 0.12. How should this be interpreted?
- The model correctly classifies 12% of cases
- The independent variables explain 12% of the variance in recidivism rates (Correct answer)
- There is a 12% probability the model is correct
- 12% of coefficients are statistically significant
Correct answer: The independent variables explain 12% of the variance in recidivism rates
RĀ² represents the proportion of variance in the dependent variable explained by the independent variables in the model.
Question 5: When a policy evaluation uses administrative records (e.g., tax filings, school enrollment data) rather than surveys, this approach is called:
- Primary data collection
- Secondary data analysis (Correct answer)
- Ethnographic research
- Experimental design
Correct answer: Secondary data analysis
Administrative records are secondary dataāinformation collected for operational purposes that researchers repurpose for analysis.
Question 6: A researcher finds a statistically significant result (p = 0.03) but the effect size (Cohen's d = 0.05) is extremely small. What is the most appropriate conclusion?
- The policy has a meaningful real-world impact
- The result is statistically significant but may lack practical significance (Correct answer)
- The study should be discarded due to low effect size
- A larger p-value should be used to confirm the finding
Correct answer: The result is statistically significant but may lack practical significance
Statistical significance and practical significance are distinct; a tiny effect can be statistically significant with a large sample but may not be meaningful for policy decisions.
Question 7: Which qualitative method involves a researcher immersing themselves in a community over an extended period to observe behaviors and practices in their natural setting?
- Structured survey
- Ethnography (Correct answer)
- Delphi method
- Regression discontinuity
Correct answer: Ethnography
Ethnography involves prolonged immersive observation in a natural setting to understand behaviors, culture, and social dynamics from an insider's perspective.
PAC Research Methodologies & Data Analysis 3
A policy analyst is studying the effect of a minimum wage increase using data from states that adopted the policy and those that did not.
This is an example of: