CAP CAP Methodology Selection & Analytics Problem Framing 2 — Questions and Answers
Question 1: Which evaluation metric is MOST appropriate for a highly imbalanced binary classification problem?
- Accuracy
- Area Under the ROC Curve (AUC-ROC) (Correct answer)
- Mean Squared Error
- R-squared
Correct answer: Area Under the ROC Curve (AUC-ROC)
AUC-ROC evaluates discrimination ability across all thresholds and is robust to class imbalance, unlike accuracy.
Question 2: A CAP analyst must choose between a linear regression model and a decision tree for a continuous outcome. The BEST reason to prefer linear regression is:
- It handles non-linear relationships better
- It provides easily interpretable coefficients for stakeholders (Correct answer)
- It requires no data preprocessing
- It always yields higher accuracy
Correct answer: It provides easily interpretable coefficients for stakeholders
Linear regression coefficients have clear directional interpretations, making results more communicable to non-technical stakeholders.
Question 3: Cross-validation is used during methodology selection primarily to:
- Speed up model training
- Estimate model performance on unseen data (Correct answer)
- Increase the training dataset size
- Reduce the number of features
Correct answer: Estimate model performance on unseen data
Cross-validation provides an unbiased estimate of how well the model will generalize by repeatedly testing on held-out data folds.
Question 4: When an analytics problem involves predicting the number of events in a fixed time interval, which model family is MOST appropriate?
- Poisson regression (Correct answer)
- Logistic regression
- Principal Component Analysis
- Naive Bayes
Correct answer: Poisson regression
Poisson regression is designed for count outcomes representing events in a fixed time or space interval.
Question 5: Which concept describes selecting the simplest model that adequately explains the data?
- Overfitting
- Parsimony (Correct answer)
- Multicollinearity
- Heteroscedasticity
Correct answer: Parsimony
Parsimony (Occam's Razor) favors simpler models to avoid overfitting and improve generalizability.
Question 6: A team debates using a neural network versus a linear model for a small dataset with 200 rows. The CAP best practice recommends:
- Always use neural networks for better accuracy
- Use the simpler linear model to avoid overfitting (Correct answer)
- Collect no additional data before deciding
- Choose based on the tool license cost
Correct answer: Use the simpler linear model to avoid overfitting
Neural networks require large datasets to generalize; on small datasets a simpler linear model typically outperforms and avoids overfitting.
CAP CAP Methodology Selection & Analytics Problem Framing 2
Which evaluation metric is MOST appropriate for a highly imbalanced binary classification problem?