CNN Training and Optimization 4 — Questions and Answers
Question 1: What is 'cosine annealing' in CNN learning rate scheduling?
- Linearly decreasing the learning rate each epoch
- Decreasing the learning rate following a cosine curve from maximum to minimum (Correct answer)
- Oscillating the learning rate between two values sinusoidally
- Multiplying the learning rate by a constant factor at fixed intervals
Correct answer: Decreasing the learning rate following a cosine curve from maximum to minimum
Cosine annealing smoothly reduces the learning rate following half a cosine cycle from an initial maximum to a minimum value, enabling gradual and smooth convergence.
Question 2: During CNN training, what is the purpose of the 'learning rate warmup' phase?
- To start with a large learning rate for fast initial progress
- To gradually increase the learning rate from near-zero before using the main schedule (Correct answer)
- To freeze early layers while training later layers first
- To reduce batch size at the start of training
Correct answer: To gradually increase the learning rate from near-zero before using the main schedule
Warmup slowly increases the learning rate from a very small value, stabilizing early training when weights and gradients are poorly scaled before the main learning rate schedule begins.
Question 3: What is 'knowledge distillation' in the context of training CNNs?
- Compressing model weights using quantization
- Training a smaller student model to mimic the outputs of a larger teacher model (Correct answer)
- Extracting feature representations for unsupervised learning
- Pruning unimportant neurons from a trained network
Correct answer: Training a smaller student model to mimic the outputs of a larger teacher model
Knowledge distillation trains a compact student CNN to reproduce the soft probability outputs of a larger pretrained teacher, transferring generalization capability to the smaller model.
Question 4: Which data augmentation technique mixes two training images and their labels by linear interpolation to improve CNN generalization?
- CutOut
- Mixup (Correct answer)
- CutMix
- AutoAugment
Correct answer: Mixup
Mixup creates virtual training samples by interpolating pixel values and labels of two random images, encouraging the model to learn more linear behavior between classes.
Question 5: What does 'gradient accumulation' allow during CNN training with limited GPU memory?
- Using a larger effective batch size by accumulating gradients over multiple mini-batches before updating (Correct answer)
- Storing gradients across epochs to avoid recomputation
- Combining gradients from multiple models for ensemble training
- Caching gradients to speed up the backward pass
Correct answer: Using a larger effective batch size by accumulating gradients over multiple mini-batches before updating
Gradient accumulation sums gradients over several small mini-batches before performing a weight update, simulating a larger batch size without requiring additional GPU memory.
Question 6: In CNN training, what is 'batch size' effect on generalization — specifically why do smaller batches often generalize better?
- Smaller batches compute more accurate gradients
- Smaller batches introduce more gradient noise acting as implicit regularization (Correct answer)
- Smaller batches converge faster to the global minimum
- Smaller batches reduce overfitting by using less data per step
Correct answer: Smaller batches introduce more gradient noise acting as implicit regularization
Small batch SGD introduces higher gradient variance (noise), which acts as implicit regularization and tends to find flatter minima that generalize better than sharp minima found by large batches.
Question 7: What is 'cyclical learning rates' (CLR) in CNN optimization?
- Decreasing the learning rate exponentially across training
- Oscillating the learning rate between a minimum and maximum bound in cycles (Correct answer)
- Using different learning rates for each layer
- Alternating between SGD and Adam optimizers
Correct answer: Oscillating the learning rate between a minimum and maximum bound in cycles
CLR varies the learning rate cyclically between a lower and upper bound, allowing the optimizer to periodically escape saddle points and explore the loss surface more effectively.
CNN Training and Optimization 4
What is 'cosine annealing' in CNN learning rate scheduling?