Deep Learning Training Techniques and Optimization 2 — Questions and Answers
Question 1: What is knowledge distillation in deep learning?
- Extracting knowledge from datasets using unsupervised methods
- Training a smaller student model to mimic the soft output probabilities of a larger teacher model (Correct answer)
- Compressing model weights through quantization
- Using ensemble predictions to label unlabeled data
Correct answer: Training a smaller student model to mimic the soft output probabilities of a larger teacher model
Knowledge distillation trains a compact student model on the soft probability outputs (dark knowledge) of a large teacher model, transferring richer information than hard labels alone.
Question 2: What is gradient accumulation used for in training large models?
- Averaging multiple loss values for stability
- Simulating a larger effective batch size by accumulating gradients over multiple mini-batches before updating weights (Correct answer)
- Storing gradients to resume interrupted training
- Clipping accumulated gradients before the optimizer step
Correct answer: Simulating a larger effective batch size by accumulating gradients over multiple mini-batches before updating weights
Gradient accumulation sums gradients over N mini-batches before performing a single optimizer step, effectively training with an N× larger batch size than GPU memory allows.
Question 3: What is mixed precision training?
- Using different optimizers for different layers
- Training with 16-bit floats for computation while maintaining 32-bit master weights for numerical stability (Correct answer)
- Mixing supervised and unsupervised objectives during training
- Using different learning rates for different parameter groups
Correct answer: Training with 16-bit floats for computation while maintaining 32-bit master weights for numerical stability
Mixed precision training performs forward and backward passes in FP16 for speed and memory savings while keeping FP32 master weights for stable gradient accumulation.
Question 4: What is the purpose of weight decay in the Adam optimizer?
- To increase the effective learning rate over time
- To regularize the model by penalizing large weight magnitudes, reducing overfitting (Correct answer)
- To decay the momentum parameter over training
- To reduce the adaptive learning rate estimates to zero
Correct answer: To regularize the model by penalizing large weight magnitudes, reducing overfitting
Weight decay adds a regularization term that shrinks weights toward zero at each update step, encouraging simpler models and reducing overfitting.
Question 5: What is cosine annealing as a learning rate schedule?
- Linearly decreasing the learning rate over training
- Reducing the learning rate following a cosine curve from an initial value to near zero over a cycle (Correct answer)
- Exponentially increasing then decreasing the learning rate
- Setting the learning rate proportional to the cosine of the batch loss
Correct answer: Reducing the learning rate following a cosine curve from an initial value to near zero over a cycle
Cosine annealing decreases the learning rate smoothly following a half-cosine curve within each cycle, often combined with warm restarts (SGDR) to escape local minima.
Question 6: What is the purpose of warm-up in learning rate scheduling for Transformer training?
- Starting with a high learning rate to converge quickly
- Gradually increasing the learning rate from near zero during the first N steps before decaying (Correct answer)
- Using a fixed learning rate for the first epoch
- Warming up the GPU before intensive training begins
Correct answer: Gradually increasing the learning rate from near zero during the first N steps before decaying
Warm-up gradually increases the learning rate from zero, stabilizing early training when model parameters and optimizer state estimates are poorly initialized.
Deep Learning Training Techniques and Optimization 2
What is knowledge distillation in deep learning?