CNN Overfitting and Regularization 5 — Questions and Answers
Question 1: Why is it problematic to use the test set to choose the best regularization hyperparameters in a CNN experiment?
- It makes training slower
- It causes information leakage, making the model appear to generalize better than it truly does (Correct answer)
- It prevents overfitting on training data
- It reduces the number of available training samples
Correct answer: It causes information leakage, making the model appear to generalize better than it truly does
Using the test set for hyperparameter selection causes data leakage, resulting in over-optimistic performance estimates that do not reflect true generalization.
Question 2: In CNN regularization, what is 'stochastic depth' and how does it reduce overfitting in deep networks?
- Randomly shortcircuiting (skipping) entire residual blocks during training (Correct answer)
- Applying variable Dropout rates to each layer
- Shrinking the depth of the network at test time
- Gradually increasing the number of layers during training
Correct answer: Randomly shortcircuiting (skipping) entire residual blocks during training
Stochastic depth randomly bypasses entire residual layers during training, effectively training an ensemble of networks with different depths.
Question 3: How does adding Gaussian noise to the input images during CNN training act as a regularizer?
- It increases the learning rate adaptively
- It forces the model to learn robust features invariant to small perturbations (Correct answer)
- It reduces the number of learnable parameters
- It ensures the loss function is strictly convex
Correct answer: It forces the model to learn robust features invariant to small perturbations
Input noise augmentation forces the CNN to learn features that are stable under small corruptions, improving robustness and generalization.
Question 4: Which metric directly measures whether a CNN is overfitting during a training run?
- The ratio of training loss to validation loss (or training accuracy gap) (Correct answer)
- The total number of parameters in the model
- The magnitude of the gradient at the final layer
- The learning rate schedule used
Correct answer: The ratio of training loss to validation loss (or training accuracy gap)
The gap between training and validation loss (or accuracy) is the direct operational metric for detecting overfitting during training.
Question 5: A CNN uses L2 regularization with coefficient λ. What happens to the gradient update rule for a weight w?
- A term −2λw is subtracted from the weight before the gradient step
- An additive penalty λ is added to the loss, and the gradient gains a +2λw term (Correct answer)
- The weight is clipped to [−λ, λ] after each update
- The gradient is divided by λ at each step
Correct answer: An additive penalty λ is added to the loss, and the gradient gains a +2λw term
L2 regularization adds λ‖w‖² to the loss, so its gradient contribution is +2λw, which is added to the usual gradient, effectively shrinking weights.
Question 6: Which augmentation strategy is specifically designed to force CNNs to use all object parts rather than the most discriminative region, by pasting patches between images?
- CutMix (Correct answer)
- Mixup
- Mosaic
- RandomCrop
Correct answer: CutMix
CutMix replaces a rectangular region of one training image with a patch from another image and blends labels proportionally, forcing the model to use the whole image.
Question 7: What is the primary reason that ensembling multiple independently trained CNNs improves generalization compared to a single model?
- Ensemble models have fewer total parameters
- Individual models overfit to different patterns; averaging cancels out their errors (Correct answer)
- Ensembles apply L1 regularization across all models automatically
- Larger ensembles always reduce training loss to zero
Correct answer: Individual models overfit to different patterns; averaging cancels out their errors
Each model in an ensemble overfits to slightly different aspects of noise in the data; averaging their predictions reduces variance and improves generalization.
CNN Overfitting and Regularization 5
Why is it problematic to use the test set to choose the best regularization hyperparameters in a CNN experiment?