CNN Overfitting and Regularization 4 — Questions and Answers
Question 1: In the context of CNNs, what does 'DropBlock' improve upon compared to standard Dropout?
- It drops contiguous blocks of spatial units, preventing nearby units from compensating for dropped ones (Correct answer)
- It applies dropout only to the final layer
- It uses a higher drop rate than standard Dropout
- It applies dropout only during backpropagation
Correct answer: It drops contiguous blocks of spatial units, preventing nearby units from compensating for dropped ones
DropBlock removes contiguous regions of feature maps so that semantic information in those regions is completely lost, making regularization more effective for CNNs.
Question 2: Which regularization strategy is most directly analogous to Bayesian model averaging for neural networks?
- L2 regularization
- Dropout at test time using MC sampling (Correct answer)
- CutOut augmentation
- Gradient clipping
Correct answer: Dropout at test time using MC sampling
Using Dropout with multiple stochastic forward passes at test time (MC Dropout) approximates Bayesian model averaging over an ensemble of models.
Question 3: When applying transfer learning to a pretrained CNN on a small target dataset, which approach BEST reduces overfitting?
- Train all layers with a high learning rate
- Freeze early convolutional layers and fine-tune only the top layers (Correct answer)
- Add more fully connected layers on top
- Remove Batch Normalization before fine-tuning
Correct answer: Freeze early convolutional layers and fine-tune only the top layers
Freezing pretrained low-level filters preserves generic features and limits trainable parameters to reduce overfitting on the small dataset.
Question 4: How does increasing the batch size during CNN training affect regularization behavior?
- Larger batches produce noisier gradients that act as regularization
- Smaller batches produce noisier gradient estimates that act as implicit regularization (Correct answer)
- Batch size has no effect on generalization
- Larger batches always improve validation accuracy
Correct answer: Smaller batches produce noisier gradient estimates that act as implicit regularization
Smaller mini-batches introduce gradient noise that acts as implicit regularization, often leading to better generalization.
Question 5: A CNN's training curve shows decreasing training loss but a validation loss that oscillates without clear improvement. What does this indicate?
- The model is underfitting and needs more capacity
- The model is overfitting and the validation set may have high variance (Correct answer)
- Gradient vanishing is present
- The learning rate is too low
Correct answer: The model is overfitting and the validation set may have high variance
Decreasing training loss alongside flat or oscillating validation loss is a sign of overfitting, possibly compounded by a small or noisy validation set.
Question 6: What role does global average pooling (GAP) play as a regularizer compared to using fully connected layers before the output?
- GAP adds more parameters to the model
- GAP eliminates the need for large fully connected layers, drastically reducing overfitting-prone parameters (Correct answer)
- GAP applies L1 regularization to the final feature map
- GAP increases gradient magnitude during backpropagation
Correct answer: GAP eliminates the need for large fully connected layers, drastically reducing overfitting-prone parameters
GAP replaces large fully connected layers by averaging each feature map to a single value, dramatically reducing parameter count and overfitting risk.
Question 7: Which scenario demonstrates 'double descent' behavior in CNN training?
- Validation loss decreases, then increases, then decreases again as model size grows beyond the interpolation threshold (Correct answer)
- Training loss increases then decreases as epochs progress
- Validation accuracy drops twice during learning rate warm-up
- Loss oscillates when two different optimizers are combined
Correct answer: Validation loss decreases, then increases, then decreases again as model size grows beyond the interpolation threshold
Double descent describes how test error can drop a second time as model size increases past the point of interpolation, counter to the classical bias-variance tradeoff.
CNN Overfitting and Regularization 4
In the context of CNNs, what does 'DropBlock' improve upon compared to standard Dropout?