CNN Training and Optimization 5 — Questions and Answers
Question 1: What is 'early stopping' in CNN training and what metric is typically monitored?
- Stopping when training loss reaches zero, monitored per batch
- Halting training when validation loss stops improving for a set number of epochs (Correct answer)
- Stopping training after a fixed number of gradient updates
- Terminating training when learning rate falls below a threshold
Correct answer: Halting training when validation loss stops improving for a set number of epochs
Early stopping halts training when the validation loss (or other metric) fails to improve for a specified patience period, preventing overfitting by avoiding unnecessary epochs.
Question 2: Which CNN training phenomenon occurs when a model performs well on training data but poorly on unseen test data?
- Underfitting
- Overfitting (Correct answer)
- Gradient saturation
- Covariate shift
Correct answer: Overfitting
Overfitting occurs when the model memorizes training data patterns rather than learning generalizable features, resulting in high training accuracy but poor test performance.
Question 3: What does 'test-time augmentation' (TTA) do to improve CNN prediction performance?
- Augments the training set with extra images at test time
- Applies multiple augmentations to test images and averages the predictions (Correct answer)
- Fine-tunes the model briefly on test data
- Increases model capacity before inference
Correct answer: Applies multiple augmentations to test images and averages the predictions
TTA generates multiple augmented versions of each test image, passes them through the model, and averages the predictions to reduce variance and improve accuracy.
Question 4: In deep CNN training, what is 'residual learning' (as in ResNets) and how does it aid optimization?
- Learning the difference from a target distribution using KL divergence
- Learning residual functions with reference to layer inputs via skip connections, easing gradient flow (Correct answer)
- Subtracting mean activations to normalize feature distributions
- Using residual blocks to reduce the number of parameters
Correct answer: Learning residual functions with reference to layer inputs via skip connections, easing gradient flow
Residual learning reformulates layers to learn residual functions F(x) added back to the input x, making identity mappings easy and allowing gradient to flow directly through skip connections to earlier layers.
Question 5: What is 'Nesterov Accelerated Gradient' (NAG) and how does it differ from standard momentum SGD?
- NAG applies momentum after the gradient step, while standard momentum applies it before
- NAG computes the gradient at the lookahead position (after the momentum step) rather than the current position (Correct answer)
- NAG adapts momentum based on curvature information
- NAG uses second-order gradient information unlike standard momentum
Correct answer: NAG computes the gradient at the lookahead position (after the momentum step) rather than the current position
NAG evaluates the gradient at the predicted future position (current weights plus momentum step) rather than the current position, providing a more informed and corrective update.
Question 6: Which loss function is most appropriate when training a CNN for multi-label classification where each image can belong to multiple classes?
- Categorical cross-entropy with softmax
- Binary cross-entropy with sigmoid applied per class (Correct answer)
- Mean squared error over class logits
- Hinge loss with one-vs-all strategy
Correct answer: Binary cross-entropy with sigmoid applied per class
Binary cross-entropy with a sigmoid activation per output neuron treats each class independently as a binary decision, correctly handling multi-label scenarios where multiple classes can be true simultaneously.
Question 7: What is 'catastrophic forgetting' in CNN training and which approach helps mitigate it?
- Loss of training data due to hardware failure; mitigated by checkpointing
- The tendency of a neural network to forget previously learned tasks when trained on new data; mitigated by Elastic Weight Consolidation (EWC) (Correct answer)
- Gradient explosion causing weight overflow; mitigated by clipping
- Forgetting to normalize inputs; mitigated by batch normalization
Correct answer: The tendency of a neural network to forget previously learned tasks when trained on new data; mitigated by Elastic Weight Consolidation (EWC)
Catastrophic forgetting occurs in continual learning when training on new tasks overwrites weights critical for old tasks; EWC adds a regularization term that penalizes changes to important weights.
CNN Training and Optimization 5
What is 'early stopping' in CNN training and what metric is typically monitored?