Data Science with Python Certification Data Science with Python Model Evaluation and Validation 4 — Questions and Answers
Question 1: Which of the following correctly describes 'precision' in a binary classification context?
- TP / (TP + FN)
- TP / (TP + FP) (Correct answer)
- TN / (TN + FP)
- TP / (TP + FP + FN)
Correct answer: TP / (TP + FP)
Precision measures the fraction of positive predictions that are actually correct: TP / (TP + FP).
Question 2: In sklearn, what does cross_val_score return by default?
- A single aggregated score
- An array of scores for each fold (Correct answer)
- The best estimator found
- The confusion matrix for each fold
Correct answer: An array of scores for each fold
cross_val_score returns a numpy array containing one score per fold, allowing you to compute statistics like mean and std.
Question 3: What is the primary advantage of nested cross-validation over standard cross-validation for model selection?
- It is computationally cheaper
- It provides an unbiased estimate of the selected model's true generalization error (Correct answer)
- It always selects the simplest model
- It eliminates the need for a test set
Correct answer: It provides an unbiased estimate of the selected model's true generalization error
Nested CV uses an outer loop for performance estimation and an inner loop for hyperparameter tuning, preventing selection bias from leaking into the performance estimate.
Question 4: A model achieves 99% accuracy on a fraud detection dataset where 99% of transactions are legitimate. What does this reveal?
- The model is excellent at detecting fraud
- Accuracy is misleading; the model likely predicts 'legitimate' for everything (Correct answer)
- The model has very low variance
- The model generalizes well to new data
Correct answer: Accuracy is misleading; the model likely predicts 'legitimate' for everything
Predicting the majority class for every sample yields 99% accuracy on this dataset, meaning the model detects zero fraudulent transactions.
Question 5: What does the 'recall' (sensitivity) metric specifically measure in binary classification?
- Fraction of predicted positives that are true positives
- Fraction of actual positives that were correctly identified (Correct answer)
- Fraction of all predictions that are correct
- Fraction of true negatives among all negatives
Correct answer: Fraction of actual positives that were correctly identified
Recall = TP / (TP + FN), measuring how well the model finds all actual positive cases in the dataset.
Question 6: In scikit-learn, which parameter of cross_val_score allows you to specify F1-score as the evaluation metric?
- metric='f1'
- scoring='f1' (Correct answer)
- criterion='f1'
- evaluator='f1'
Correct answer: scoring='f1'
The scoring parameter accepts string metric names like 'f1', 'f1_macro', or 'roc_auc' to control what cross_val_score computes.
Question 7: What is 'model calibration' and why does it matter in classification?
- Training a model until it converges
- Ensuring predicted probabilities match empirical frequencies of outcomes (Correct answer)
- Scaling features before model training
- Adjusting decision boundary thresholds post-training
Correct answer: Ensuring predicted probabilities match empirical frequencies of outcomes
A calibrated model's predicted probability of 0.7 means the event truly occurs about 70% of the time — critical when probabilities drive business decisions.
Data Science with Python Certification Data Science with Python Model Evaluation and Validation 4
Which of the following correctly describes 'precision' in a binary classification context?