Statistical Concepts and Analysis Flashcards

A data scientist runs an A/B test on a new website feature and calculates a p-value of 0.03. The significance level (alpha) was set at 0.05. Which of the following is the most accurate interpretation of this result?

Answer: The result is statistically significant, providing evidence to reject the null hypothesis.

The p-value (0.03) is less than the pre-determined significance level, alpha (0.05). This indicates a statistically significant result. Therefore, we reject the null hypothesis, which typically states there is no difference or no effect. The p-value itself is the probability of observing the data (or more extreme data) if the null hypothesis were true, not the probability that the null hypothesis is true.

In the context of the bias-variance tradeoff in machine learning, a model with high variance and low bias is most likely to exhibit which of the following?

Answer: Overfitting, where it performs very well on training data but poorly on test data.

High variance indicates that the model is highly sensitive to small fluctuations in the training data, essentially memorizing the noise along with the signal. This leads to excellent performance on the training set but poor generalization to new, unseen data (test set), a classic sign of overfitting. Low bias means the model is complex enough to capture the underlying patterns.

A pharmaceutical company is testing a new drug. The null hypothesis is that the drug has no effect. A Type I error in this scenario would be:

Answer: Concluding the drug is effective when it actually is not.

A Type I error occurs when we incorrectly reject a true null hypothesis. In this case, the null hypothesis is that the drug has no effect. Rejecting this true null hypothesis means concluding the drug is effective when, in reality, it isn't. This is also known as a 'false positive'.

A pollster wants to estimate the average income of adults in a large city. They take a random sample of 500 adults. According to the Central Limit Theorem, what can be said about the distribution of the sample mean income?

Answer: It will be approximately a normal distribution, regardless of the city's actual income distribution.

The Central Limit Theorem (CLT) states that the distribution of sample means will be approximately normal for a sufficiently large sample size, even if the original population distribution is not normal. This principle is fundamental for making inferences about a population mean from a sample mean.

A data analyst calculates a 95% confidence interval for the average user engagement time on a new app feature to be [15.2 minutes, 18.6 minutes]. What is the correct interpretation of this interval?

Answer: If we were to repeat this sampling process many times, 95% of the calculated confidence intervals would contain the true average engagement time for all users.

A 95% confidence interval has a frequentist interpretation. It means that the method used to construct the interval will, in the long run, capture the true population parameter (in this case, the true average engagement time) in 95% of the samples taken. It does not give the probability that a single, specific interval contains the true mean.

In an A/B test comparing two website designs (A and B), the goal is to determine which design leads to a higher conversion rate. What is the primary purpose of determining statistical significance in this context?

Answer: To ensure the observed difference between the designs is due to their effectiveness and not random chance.

Statistical significance in A/B testing is used to assess the likelihood that the observed difference in performance between two versions is not just a result of random noise or sampling error. By achieving statistical significance, we gain confidence that the observed effect is real and attributable to the changes made.