Deep Learning Training Techniques and Optimization 2

Question 1

What is knowledge distillation in deep learning?

Accepted Answer

Training a smaller student model to mimic the soft output probabilities of a larger teacher model

Answer

Extracting knowledge from datasets using unsupervised methods

Answer

Compressing model weights through quantization

Answer

Using ensemble predictions to label unlabeled data

Question 2

What is gradient accumulation used for in training large models?

Accepted Answer

Simulating a larger effective batch size by accumulating gradients over multiple mini-batches before updating weights

Answer

Averaging multiple loss values for stability

Answer

Storing gradients to resume interrupted training

Answer

Clipping accumulated gradients before the optimizer step

Question 3

What is mixed precision training?

Accepted Answer

Training with 16-bit floats for computation while maintaining 32-bit master weights for numerical stability

Answer

Using different optimizers for different layers

Answer

Mixing supervised and unsupervised objectives during training

Answer

Using different learning rates for different parameter groups

Question 4

What is the purpose of weight decay in the Adam optimizer?

Accepted Answer

To regularize the model by penalizing large weight magnitudes, reducing overfitting

Answer

To increase the effective learning rate over time

Answer

To decay the momentum parameter over training

Answer

To reduce the adaptive learning rate estimates to zero

Question 5

What is cosine annealing as a learning rate schedule?

Accepted Answer

Reducing the learning rate following a cosine curve from an initial value to near zero over a cycle

Answer

Linearly decreasing the learning rate over training

Answer

Exponentially increasing then decreasing the learning rate

Answer

Setting the learning rate proportional to the cosine of the batch loss

Question 6

What is the purpose of warm-up in learning rate scheduling for Transformer training?

Accepted Answer

Gradually increasing the learning rate from near zero during the first N steps before decaying

Answer

Starting with a high learning rate to converge quickly

Answer

Using a fixed learning rate for the first epoch

Answer

Warming up the GPU before intensive training begins