Statistical Concepts and Inference Flashcards

A data scientist conducts an A/B test for a new website feature and calculates a p-value of 0.03. The chosen significance level (alpha) is 0.05. How should the p-value be interpreted?

Answer: There is a 3% probability of observing the current result, or a more extreme one, if the null hypothesis is true.

The p-value represents the probability of obtaining the observed results, or more extreme results, assuming the null hypothesis is true. Since 0.03 is less than the significance level of 0.05, the result is statistically significant, leading to the rejection of the null hypothesis. The p-value does not indicate the probability of the null hypothesis being true or the magnitude of the effect.

A machine learning model is developed to predict customer churn. In testing, it is observed that the model performs exceptionally well on the training data but poorly on new, unseen data. Which of the following concepts does this scenario best describe?

Answer: The bias-variance tradeoff

This scenario is a classic example of the bias-variance tradeoff, specifically overfitting. The model has low bias, as it fits the training data very well, but high variance, as it is too sensitive to the training data and fails to generalize to new data. The tension between these two sources of error is known as the bias-variance tradeoff.

A researcher calculates a 95% confidence interval for the average height of a certain population to be [170 cm, 180 cm]. Which of the following is the correct interpretation of this interval?

Answer: If we were to repeat the sampling process many times, 95% of the calculated confidence intervals would contain the true population mean.

A 95% confidence interval means that if the same sampling procedure were repeated numerous times, 95% of the resulting confidence intervals would be expected to contain the true population parameter. It is a statement about the reliability of the estimation procedure, not a direct probability statement about the true population mean falling within a specific interval.

In the context of hypothesis testing, what is a Type II error?

Answer: Failing to reject the null hypothesis when it is actually false.

A Type II error, also known as a false negative, occurs when one fails to reject a null hypothesis that is actually false. In other words, a real effect or difference exists, but the statistical test was not sensitive enough to detect it. Rejecting a true null hypothesis is a Type I error.

According to the Central Limit Theorem, what happens to the sampling distribution of the sample mean as the sample size increases, regardless of the population's original distribution?

Answer: It approaches a normal distribution.

The Central Limit Theorem (CLT) is a fundamental principle stating that, for a sufficiently large sample size, the sampling distribution of the sample mean will be approximately normally distributed, regardless of the shape of the population's distribution. This allows for statistical inference even when the underlying data is not normally distributed.

A company's data science team wants to test if a new marketing campaign led to a higher average daily website visit duration. The null hypothesis (H₀) is that the average duration is unchanged or lower, while the alternative hypothesis (H₁) is that the average duration is higher. After analysis, they fail to reject the null hypothesis. However, the campaign did, in fact, increase the average duration. What type of error has been made?

Answer: Type II Error

This scenario describes a Type II error. A Type II error occurs when you fail to reject a null hypothesis that is actually false. Here, the null hypothesis (no increase in duration) was not rejected, but in reality, it was false (the campaign did increase duration). This is also known as a 'false negative'.