CNN Training and Optimization 2 — Questions and Answers
Question 1: What does the learning rate scheduler 'ReduceLROnPlateau' do when validation loss stops improving?
- Increases the learning rate by a factor
- Reduces the learning rate by a factor (Correct answer)
- Resets the model weights
- Stops training immediately
Correct answer: Reduces the learning rate by a factor
ReduceLROnPlateau monitors a metric and reduces the learning rate by a specified factor when no improvement is seen for a patience number of epochs.
Question 2: Which technique adds Gaussian noise to the gradients during CNN training to help escape sharp local minima?
- Gradient clipping
- Weight decay
- Stochastic gradient noise (Correct answer)
- Batch normalization
Correct answer: Stochastic gradient noise
Stochastic gradient noise injects Gaussian noise into gradients during optimization, helping the model escape sharp local minima and find flatter, more generalizable optima.
Question 3: In CNN training, what does 'warm restarts' refer to in the context of learning rate schedules?
- Restarting training from scratch periodically
- Periodically resetting the learning rate to a high value then annealing it (Correct answer)
- Warming up GPU memory before training
- Gradually increasing batch size
Correct answer: Periodically resetting the learning rate to a high value then annealing it
Warm restarts (SGDR) periodically resets the learning rate to a maximum value and then follows a cosine annealing schedule, helping the model explore different loss landscape regions.
Question 4: What is the primary benefit of using mixed-precision training (FP16 + FP32) in CNN optimization?
- Improved model accuracy
- Reduced training time and memory usage with minimal accuracy loss (Correct answer)
- Elimination of gradient vanishing
- Automatic hyperparameter tuning
Correct answer: Reduced training time and memory usage with minimal accuracy loss
Mixed-precision training uses 16-bit floats for most operations and 32-bit for critical accumulations, significantly reducing memory footprint and speeding up computation on modern GPUs.
Question 5: During CNN training, which regularization method randomly drops entire feature maps rather than individual neurons?
- Dropout
- DropBlock (Correct answer)
- Weight decay
- SpatialDropout
Correct answer: DropBlock
DropBlock drops contiguous regions of feature maps, which is more effective for CNNs than standard dropout because adjacent units in feature maps tend to be correlated.
Question 6: What problem does gradient checkpointing solve in CNN training?
- Vanishing gradients in deep networks
- High memory usage during backpropagation by recomputing activations (Correct answer)
- Exploding gradients during training
- Slow convergence due to poor initialization
Correct answer: High memory usage during backpropagation by recomputing activations
Gradient checkpointing reduces memory by not storing all intermediate activations; instead, it recomputes them during the backward pass, trading compute time for memory savings.
Question 7: Which optimizer introduces a per-parameter adaptive learning rate based on the second moment of gradients?
- SGD with momentum
- Adagrad
- Adam (Correct answer)
- RMSprop
Correct answer: Adam
Adam maintains both first moment (mean) and second moment (uncentered variance) estimates of gradients, providing adaptive per-parameter learning rates with bias correction.
CNN Training and Optimization 2
What does the learning rate scheduler 'ReduceLROnPlateau' do when validation loss stops improving?