Software Testing Research & Evidence-Based Practice 5 — Questions and Answers
Question 1: What is 'gray literature' in the context of software testing evidence gathering?
- Non-peer-reviewed sources such as technical reports, blogs, and industry white papers (Correct answer)
- Low-quality peer-reviewed papers that failed the review process
- Research involving color-coded test case priority schemes
- Defect data collected from legacy or undocumented systems
Correct answer: Non-peer-reviewed sources such as technical reports, blogs, and industry white papers
Gray literature includes practitioner knowledge published outside formal academic channels and is valuable for capturing industrial practice.
Question 2: In a software testing experiment comparing two code coverage criteria, what is the 'dependent variable'?
- The outcome being measured, such as defect detection rate or branch coverage achieved (Correct answer)
- The coverage criterion chosen for each group
- The programming language used to write the software under test
- The number of testers assigned to each group
Correct answer: The outcome being measured, such as defect detection rate or branch coverage achieved
The dependent variable is what is measured as the outcome; the independent variable is the coverage criterion being manipulated.
Question 3: What does 'inter-rater reliability' measure in a qualitative software testing study?
- The degree to which independent raters agree when classifying or coding the same data (Correct answer)
- The consistency of automated test execution across multiple CI pipelines
- The repeatability of exploratory testing sessions by the same tester
- The reproducibility of performance benchmark results across runs
Correct answer: The degree to which independent raters agree when classifying or coding the same data
High inter-rater reliability indicates that different researchers reach the same conclusions when analyzing qualitative data, strengthening credibility.
Question 4: A meta-analysis of unit testing studies reports a 'weighted mean effect size.' Why is weighting applied?
- To give larger or more precise studies more influence on the combined estimate (Correct answer)
- To normalize defect counts across projects with different programming languages
- To penalize studies that did not use random assignment
- To adjust for inflation in lines-of-code counts over different publication years
Correct answer: To give larger or more precise studies more influence on the combined estimate
Weighting ensures that higher-quality or larger studies contribute more to the pooled effect size estimate than small, imprecise studies.
Question 5: What is the primary risk of 'HARKing' (Hypothesizing After Results are Known) in software testing research?
- It presents post-hoc observations as pre-planned hypotheses, inflating false-positive rates (Correct answer)
- It prevents researchers from publishing negative results
- It limits the scope of a study to a single programming language
- It forces the use of parametric statistics on non-normal data
Correct answer: It presents post-hoc observations as pre-planned hypotheses, inflating false-positive rates
HARKing misrepresents exploratory findings as confirmatory, bypassing the protection against chance results that pre-registration provides.
Question 6: Which EBSE (Evidence-Based Software Engineering) step involves critically appraising study quality before applying findings?
- Evidence evaluation (Correct answer)
- Evidence aggregation
- Question formulation
- Evidence dissemination
Correct answer: Evidence evaluation
Evidence evaluation assesses the rigor, relevance, and risk of bias in studies before their conclusions are used to guide practice.
Question 7: A practitioner reads that 'automated UI testing reduces regression time by 40%.' What follow-up question is most aligned with evidence-based practice?
- What was the study context, sample size, and comparison baseline that produced that figure? (Correct answer)
- Which vendor's tool was used and can the same tool be licensed immediately?
- How many UI screens does the application have that could benefit?
- Was the 40% measured in hours or in percentage of sprint capacity?
Correct answer: What was the study context, sample size, and comparison baseline that produced that figure?
Evidence-based practice requires understanding the study's context and methodology to judge whether the finding applies to one's own situation.
Software Testing Research & Evidence-Based Practice 5
What is 'gray literature' in the context of software testing evidence gathering?