Software Testing Research & Evidence-Based Practice 4 — Questions and Answers
Question 1: Which statistical test is most appropriate for comparing defect counts between two independent test techniques with non-normal distributions?
- Mann-Whitney U test (Correct answer)
- Paired t-test
- Chi-square goodness-of-fit test
- Pearson correlation coefficient
Correct answer: Mann-Whitney U test
The Mann-Whitney U test is a non-parametric alternative to the t-test, suitable for comparing two independent groups when normality cannot be assumed.
Question 2: In evidence-based testing, what does the PICO framework stand for?
- Population, Intervention, Comparison, Outcome (Correct answer)
- Protocol, Implementation, Coverage, Output
- Process, Integration, Complexity, Observability
- Plan, Investigate, Confirm, Optimize
Correct answer: Population, Intervention, Comparison, Outcome
PICO structures research questions by defining the Population, Intervention being studied, Comparison condition, and desired Outcome.
Question 3: What is 'construct validity' in the context of a software testing research study?
- Whether the measurement instruments actually capture the theoretical concept they claim to measure (Correct answer)
- Whether the test suite covers all code paths in the system under test
- Whether the study's conclusions are supported by the data collected
- Whether the software product meets its functional requirements
Correct answer: Whether the measurement instruments actually capture the theoretical concept they claim to measure
Construct validity ensures that the measures used (e.g., fault-detection rate) truly represent the theoretical construct (e.g., test effectiveness).
Question 4: A researcher claims that TDD reduces defect density based on a study of 10 student projects. What threat to validity is most prominent?
- Low external validity due to a non-representative student population (Correct answer)
- High internal validity due to randomization
- Construct validity failure due to missing mutation testing
- Instrumentation bias from using automated test runners
Correct answer: Low external validity due to a non-representative student population
Student projects differ significantly from industrial projects in scale, domain, and developer experience, limiting generalizability.
Question 5: What distinguishes a 'mapping study' from a full 'systematic literature review' in software testing research?
- A mapping study provides a broad overview of research trends; an SLR provides in-depth synthesis of evidence (Correct answer)
- A mapping study is more rigorous because it uses stricter inclusion criteria
- A mapping study always includes meta-analysis; an SLR does not
- An SLR covers only published conference papers; a mapping study includes gray literature
Correct answer: A mapping study provides a broad overview of research trends; an SLR provides in-depth synthesis of evidence
Mapping studies classify and visualize the landscape of research on a topic, while SLRs deeply synthesize evidence to answer specific research questions.
Question 6: In a defect prediction model research study, what is 'data leakage' and why is it harmful?
- When future data is inadvertently used during model training, inflating performance metrics (Correct answer)
- When test data is accidentally committed to the source repository
- When defect reports from one project contaminate another project's dataset
- When production logs are shared with the QA team before system release
Correct answer: When future data is inadvertently used during model training, inflating performance metrics
Data leakage causes artificially high accuracy by allowing the model to 'see' information it would not have at prediction time.
Question 7: Which of the following best illustrates the application of 'evidence-based practice' by a test manager?
- Adopting code review because controlled studies show it reduces defect escape rates, rather than following intuition (Correct answer)
- Requiring the team to follow whichever technique was taught in the manager's last training course
- Choosing a test tool based on vendor marketing materials and cost alone
- Applying a testing technique because a respected colleague recommended it
Correct answer: Adopting code review because controlled studies show it reduces defect escape rates, rather than following intuition
Evidence-based practice means grounding decisions in empirical research findings rather than tradition, authority, or intuition alone.
Software Testing Research & Evidence-Based Practice 4
Which statistical test is most appropriate for comparing defect counts between two independent test techniques with non-normal distributions?