Deep Learning Training Techniques and Optimization 2

Question 1

What is knowledge distillation in deep learning?

Accepted Answer

Training a smaller student model to mimic the soft output probabilities of a larger teacher model

Answer

Knowledge distillation trains a compact student model on the soft probability outputs (dark knowledge) of a large teacher model, transferring richer information than hard labels alone.

Question 2

What is gradient accumulation used for in training large models?

Accepted Answer

Simulating a larger effective batch size by accumulating gradients over multiple mini-batches before updating weights

Answer

Gradient accumulation sums gradients over N mini-batches before performing a single optimizer step, effectively training with an N× larger batch size than GPU memory allows.

Question 3

What is mixed precision training?

Accepted Answer

Training with 16-bit floats for computation while maintaining 32-bit master weights for numerical stability

Answer

Mixed precision training performs forward and backward passes in FP16 for speed and memory savings while keeping FP32 master weights for stable gradient accumulation.

Question 4

What is the purpose of weight decay in the Adam optimizer?

Accepted Answer

To regularize the model by penalizing large weight magnitudes, reducing overfitting

Answer

Weight decay adds a regularization term that shrinks weights toward zero at each update step, encouraging simpler models and reducing overfitting.

Question 5

What is cosine annealing as a learning rate schedule?

Accepted Answer

Reducing the learning rate following a cosine curve from an initial value to near zero over a cycle

Answer

Cosine annealing decreases the learning rate smoothly following a half-cosine curve within each cycle, often combined with warm restarts (SGDR) to escape local minima.

Deep Learning Practice Test

Deep Learning Practice Test

Deep Learning Training Techniques and Optimization 2