Web Traffic Research & Evidence-Based Practice 3 — Questions and Answers
Question 1: An analyst notices that a competitor launched a major campaign on the same day as an internal SEO update. Which research concept describes this problem?
- Selection bias
- Confounding (Correct answer)
- Attrition bias
- Publication bias
Correct answer: Confounding
Confounding occurs when a third variable (the competitor campaign) is associated with both the treatment and the outcome, making it impossible to isolate the SEO update's effect.
Question 2: When segmenting traffic data by device type for a research study, which sampling approach ensures proportional representation of mobile, desktop, and tablet users?
- Simple random sampling
- Stratified random sampling (Correct answer)
- Convenience sampling
- Cluster sampling
Correct answer: Stratified random sampling
Stratified random sampling divides the population into device-type strata and samples from each proportionally, ensuring all segments are adequately represented.
Question 3: A study reports that pages with longer dwell time rank higher in search results. What causal inference problem must researchers address before concluding dwell time improves rankings?
- Measurement error
- Reverse causality (Correct answer)
- Attrition
- Floor effect
Correct answer: Reverse causality
Reverse causality is the problem that higher-ranking pages attract more engaged users, so dwell time may be a consequence of ranking rather than a cause.
Question 4: Which visualization is most effective for communicating the distribution of session durations across thousands of user visits in a research report?
- Pie chart
- Violin plot (Correct answer)
- Stacked bar chart
- Line graph
Correct answer: Violin plot
A violin plot combines a box plot with a kernel density estimate, revealing the full shape of the session duration distribution including skewness and multimodal patterns.
Question 5: A researcher uses Google Analytics data to study user behavior. Which limitation must be disclosed when publishing findings?
- GA data is only available for paid sites
- GA sampling occurs for high-traffic reports, potentially biasing estimates (Correct answer)
- GA cannot track mobile users
- GA data is deleted after 30 days by default
Correct answer: GA sampling occurs for high-traffic reports, potentially biasing estimates
Google Analytics applies data sampling in standard reports for high-traffic properties, meaning reported metrics may not reflect all sessions, introducing bias.
Question 6: In a web traffic experiment, you randomly assign 50% of landing page visitors to a new design. After 2 weeks you find p=0.04 favoring the new design. What additional factor should you report?
- The number of developers who built the new design
- Statistical power and the minimum detectable effect size (Correct answer)
- Whether the old design used images
- The server response time of each variant
Correct answer: Statistical power and the minimum detectable effect size
Reporting statistical power and minimum detectable effect confirms the study was adequately sized to detect meaningful differences, not just marginally significant ones.
Question 7: Which method should researchers use to handle missing traffic data caused by Google Analytics outages in a longitudinal study?
- Delete all rows with missing values
- Multiple imputation using surrounding time-period values (Correct answer)
- Replace missing values with zero
- Repeat the last observed value indefinitely
Correct answer: Multiple imputation using surrounding time-period values
Multiple imputation generates plausible values for missing data based on observed patterns, preserving the dataset's statistical properties better than deletion or zero-filling.
Web Traffic Research & Evidence-Based Practice 3
An analyst notices that a competitor launched a major campaign on the same day as an internal SEO update.
Which research concept describes this problem?