Deep Learning Training Techniques and Optimization 1 — Questions and Answers
Question 1: What is the Adam optimizer and what are its two main components?
- A variant of SGD with only momentum
- An optimizer combining adaptive learning rates (RMSProp) and momentum (first moment) with bias correction (Correct answer)
- A second-order optimization method using the Hessian
- An optimizer that applies gradient clipping automatically
Correct answer: An optimizer combining adaptive learning rates (RMSProp) and momentum (first moment) with bias correction
Adam maintains per-parameter estimates of both the first moment (mean of gradients) and second moment (uncentered variance), using them to compute adaptive learning rates with bias correction.
Question 2: What is L2 regularization (weight decay) and how does it combat overfitting?
- Adding a penalty proportional to the absolute value of weights to the loss
- Adding a penalty proportional to the squared magnitude of weights to the loss, discouraging large weights (Correct answer)
- Dropping random neurons during training
- Normalizing activations between layers
Correct answer: Adding a penalty proportional to the squared magnitude of weights to the loss, discouraging large weights
L2 regularization adds λ∑w² to the loss, penalizing large weights and encouraging the model to use smaller, more distributed parameter values that generalize better.
Question 3: What is the purpose of a learning rate scheduler during training?
- Setting the learning rate based on validation loss only
- Adjusting the learning rate over training time, typically decreasing it to enable fine-grained convergence (Correct answer)
- Choosing the optimizer algorithm dynamically
- Scaling the learning rate by the number of layers
Correct answer: Adjusting the learning rate over training time, typically decreasing it to enable fine-grained convergence
Learning rate schedulers modify the learning rate during training — common strategies include step decay, cosine annealing, and warmup — to balance initial progress with final precision.
Question 4: What is early stopping as a regularization technique?
- Stopping training when training loss reaches zero
- Halting training when validation performance stops improving to prevent overfitting (Correct answer)
- Removing neurons after a fixed number of epochs
- Reducing the learning rate after each epoch automatically
Correct answer: Halting training when validation performance stops improving to prevent overfitting
Early stopping monitors validation loss and terminates training when it stops decreasing, saving the model checkpoint with the best generalization before overfitting occurs.
Question 5: What distinguishes overfitting from underfitting in a deep learning model?
- Overfitting: high training and test error; underfitting: low training error, high test error
- Overfitting: low training error, high test error; underfitting: high training and test error (Correct answer)
- Overfitting: model is too simple; underfitting: model is too complex
- They describe the same phenomenon from different perspectives
Correct answer: Overfitting: low training error, high test error; underfitting: high training and test error
Overfitting occurs when a model memorizes training data, achieving low training error but poor generalization (high test error); underfitting occurs when the model is too simple to capture the data's patterns.
Question 6: What is the purpose of a validation set in model training?
- Training model weights by gradient descent
- Providing an unbiased estimate of generalization during training for hyperparameter tuning (Correct answer)
- Serving as the final evaluation set reported to stakeholders
- Augmenting the training set with additional examples
Correct answer: Providing an unbiased estimate of generalization during training for hyperparameter tuning
The validation set evaluates model performance during training to guide decisions like early stopping, hyperparameter tuning, and architecture selection, without touching the held-out test set.
Deep Learning Training Techniques and Optimization 1
What is the Adam optimizer and what are its two main components?