Software Testing Research & Evidence-Based Practice 3 — Questions and Answers
Question 1: Which research method involves observing software teams in their natural work environment without manipulating variables?
- Ethnographic study (Correct answer)
- Controlled experiment
- Meta-analysis
- Randomized controlled trial
Correct answer: Ethnographic study
Ethnographic studies observe practitioners in their real environment to understand behaviors and practices without researcher intervention.
Question 2: When a testing research paper reports a 'p-value < 0.05', what does this conventionally indicate?
- The result is statistically significant under the null hypothesis at a 5% threshold (Correct answer)
- The test suite achieved 95% branch coverage
- Five percent of the bugs remain undetected
- The study has a 5% chance of being wrong overall
Correct answer: The result is statistically significant under the null hypothesis at a 5% threshold
A p-value below 0.05 means there is less than a 5% probability of observing the result if the null hypothesis were true, meeting the conventional significance threshold.
Question 3: What is 'publication bias' and why is it relevant to evidence-based software testing?
- The tendency for positive results to be published more often, skewing the available evidence (Correct answer)
- A bias introduced when testers know which version of code they are testing
- The preference of journals for papers with large datasets
- A systematic error in code coverage measurement tools
Correct answer: The tendency for positive results to be published more often, skewing the available evidence
Publication bias inflates apparent effectiveness of techniques because negative or null results are less likely to be published.
Question 4: In a survey-based software testing study, what is 'response bias'?
- Participants answer in ways they believe are expected rather than reflecting their true experience (Correct answer)
- The survey tool crashes before all responses are recorded
- Respondents misunderstand technical terms in the questionnaire
- Participants who respond are not representative of the target population
Correct answer: Participants answer in ways they believe are expected rather than reflecting their true experience
Response bias occurs when participants skew answers toward socially desirable or expected responses rather than honest ones.
Question 5: What does 'effect size' measure in a software testing experiment, and why does it matter beyond p-values?
- The practical magnitude of a difference between groups, independent of sample size (Correct answer)
- The percentage of test cases that passed in the experimental group
- The number of defects fixed during the study period
- The processing time saved by the new testing technique
Correct answer: The practical magnitude of a difference between groups, independent of sample size
Effect size quantifies how large a difference is in practical terms, whereas a p-value only indicates whether it is statistically distinguishable.
Question 6: Which of the following is a key advantage of using 'case studies' in software testing research?
- They capture rich contextual details from real industrial projects (Correct answer)
- They guarantee causality through random assignment
- They produce results generalizable to all software projects
- They require only a small amount of researcher time to complete
Correct answer: They capture rich contextual details from real industrial projects
Case studies provide in-depth, contextual insights from actual projects, revealing practical nuances that controlled experiments may miss.
Question 7: What is the role of a 'research protocol' in a systematic mapping study of software testing techniques?
- It documents the search strategy, inclusion/exclusion criteria, and data extraction process before the study begins (Correct answer)
- It defines the API contracts for automated test execution
- It specifies the hardware configuration for running performance tests
- It outlines the code review process for the software under test
Correct answer: It documents the search strategy, inclusion/exclusion criteria, and data extraction process before the study begins
A research protocol pre-specifies the methodology to ensure reproducibility and reduce researcher bias in systematic mapping studies.
Software Testing Research & Evidence-Based Practice 3
Which research method involves observing software teams in their natural work environment without manipulating variables?