Data Science with Python Certification Data Science with Python Model Evaluation and Validation 3 — Questions and Answers
Question 1: In regression model evaluation, what does a residual plot with a clear funnel shape (heteroscedasticity) suggest?
- The model perfectly fits the data
- Residual variance is constant across predicted values
- Residual variance increases with predicted values, violating OLS assumptions (Correct answer)
- The model is underfitting
Correct answer: Residual variance increases with predicted values, violating OLS assumptions
A funnel-shaped residual plot indicates heteroscedasticity — the error variance is not constant, violating a key linear regression assumption.
Question 2: Which scikit-learn utility allows you to chain preprocessing steps and a model estimator so that cross-validation applies transformations correctly within each fold?
- ColumnTransformer
- Pipeline (Correct answer)
- FeatureUnion
- TransformedTargetRegressor
Correct answer: Pipeline
sklearn.pipeline.Pipeline ensures that fit/transform steps are applied only to training data within each fold, preventing data leakage.
Question 3: What is the bias-variance tradeoff implication of increasing model complexity on a fixed dataset?
- Bias increases and variance decreases
- Both bias and variance decrease
- Bias decreases and variance increases (Correct answer)
- Both bias and variance increase
Correct answer: Bias decreases and variance increases
More complex models fit training data more closely (lower bias) but become sensitive to noise, increasing variance and risking overfitting.
Question 4: Which metric penalizes large prediction errors more heavily due to squaring residuals?
- Mean Absolute Error (MAE)
- Mean Squared Error (MSE) (Correct answer)
- Median Absolute Deviation (MAD)
- Mean Absolute Percentage Error (MAPE)
Correct answer: Mean Squared Error (MSE)
MSE squares each residual before averaging, so large errors contribute disproportionately more to the final score than small errors.
Question 5: In the context of model evaluation, what is 'overfitting' most precisely?
- Low training error and low test error
- High training error and high test error
- Low training error and high test error (Correct answer)
- High training error and low test error
Correct answer: Low training error and high test error
An overfit model memorizes training data, achieving low training error but failing to generalize, resulting in high test error.
Question 6: When comparing multiple models using cross-validation scores, why is it important to check the standard deviation of the CV scores, not just the mean?
- Standard deviation determines the optimal number of folds
- High standard deviation indicates unstable model performance across folds (Correct answer)
- Standard deviation is used to select hyperparameters
- It determines whether to use stratification
Correct answer: High standard deviation indicates unstable model performance across folds
A high standard deviation across folds indicates the model is sensitive to the particular data split, suggesting unstable or unreliable performance.
Question 7: Which evaluation approach is most appropriate when the dataset is very small (e.g., fewer than 100 samples)?
- Hold-out validation with 80/20 split
- 10-fold cross-validation
- Leave-One-Out Cross-Validation (LOOCV) (Correct answer)
- Train on full dataset without validation
Correct answer: Leave-One-Out Cross-Validation (LOOCV)
LOOCV uses n-1 samples for training in each iteration, maximizing training data usage — critical when the overall dataset is very small.
Data Science with Python Certification Data Science with Python Model Evaluation and Validation 3
In regression model evaluation, what does a residual plot with a clear funnel shape (heteroscedasticity) suggest?