MS-DS Master of Data science Machine Learning 3 — Questions and Answers
Question 1: What does the VC dimension measure in statistical learning theory?
- The number of training samples needed for convergence
- The capacity or complexity of a hypothesis class, defined by the largest set it can shatter (Correct answer)
- The margin width between support vectors in SVMs
- The variance of the model's predictions across datasets
Correct answer: The capacity or complexity of a hypothesis class, defined by the largest set it can shatter
The VC dimension is the size of the largest set of points that a hypothesis class can classify correctly in all possible ways, quantifying model capacity.
Question 2: In principal component analysis (PCA), what do the principal components represent?
- The original features sorted by their correlation with the target
- Orthogonal directions of maximum variance in the data (Correct answer)
- Cluster centroids in the reduced feature space
- Eigenvectors of the covariance matrix sorted by eigenvalue in ascending order
Correct answer: Orthogonal directions of maximum variance in the data
PCA finds orthogonal axes (principal components) that capture the directions of maximum variance, sorted from highest to lowest explained variance.
Question 3: Which cross-validation strategy is most appropriate when the dataset has a temporal ordering?
- Stratified k-fold
- Leave-one-out
- Time-series split (walk-forward validation) (Correct answer)
- Monte Carlo cross-validation
Correct answer: Time-series split (walk-forward validation)
Time-series split trains on past observations and validates on future ones, preserving temporal order and preventing data leakage from future to past.
Question 4: In the Expectation-Maximization (EM) algorithm for Gaussian Mixture Models, what happens in the E-step?
- Model parameters (means, covariances) are updated
- Each data point is hard-assigned to its nearest centroid
- Soft responsibilities (posterior probabilities of cluster membership) are computed for each point (Correct answer)
- The number of components is selected by BIC
Correct answer: Soft responsibilities (posterior probabilities of cluster membership) are computed for each point
The E-step computes the expected cluster membership (responsibility) of each data point given the current model parameters using Bayes' theorem.
Question 5: What does SMOTE (Synthetic Minority Oversampling TEchnique) do to address class imbalance?
- Removes majority class samples near the decision boundary
- Creates synthetic minority samples by interpolating between existing minority instances (Correct answer)
- Assigns higher misclassification costs to the majority class
- Resamples both classes to equal size by random duplication
Correct answer: Creates synthetic minority samples by interpolating between existing minority instances
SMOTE generates new synthetic minority samples by selecting a minority point and interpolating along the line segment to one of its k nearest minority neighbors.
Question 6: Which metric is most informative when evaluating a classifier on a highly imbalanced dataset where false negatives are costly?
- Accuracy
- Recall (Sensitivity) (Correct answer)
- Specificity
- Cohen's Kappa
Correct answer: Recall (Sensitivity)
Recall measures the proportion of actual positives correctly identified; when false negatives are costly and classes are imbalanced, accuracy is misleading and recall directly captures missed positives.
Question 7: In a neural network, what is the vanishing gradient problem?
- Gradients explode to very large values during backpropagation
- Gradients become extremely small in early layers, slowing or halting learning (Correct answer)
- Weight updates are applied before the full batch is processed
- Neurons become permanently inactive due to dead ReLU units
Correct answer: Gradients become extremely small in early layers, slowing or halting learning
During backpropagation through many layers with saturating activations (e.g., sigmoid), gradients are repeatedly multiplied by small values, shrinking exponentially and causing early layers to learn very slowly.
MS-DS Master of Data science Machine Learning 3
What does the VC dimension measure in statistical learning theory?