Deep Learning Training Techniques and Optimization

Question 1

What is the Adam optimizer and what are its two main components?

Accepted Answer

An optimizer combining adaptive learning rates (RMSProp) and momentum (first moment) with bias correction

Answer

Adam maintains per-parameter estimates of both the first moment (mean of gradients) and second moment (uncentered variance), using them to compute adaptive learning rates with bias correction.

Question 2

What is L2 regularization (weight decay) and how does it combat overfitting?

Accepted Answer

Adding a penalty proportional to the squared magnitude of weights to the loss, discouraging large weights

Answer

L2 regularization adds λ∑w² to the loss, penalizing large weights and encouraging the model to use smaller, more distributed parameter values that generalize better.

Question 3

What is the purpose of a learning rate scheduler during training?

Accepted Answer

Adjusting the learning rate over training time, typically decreasing it to enable fine-grained convergence

Answer

Learning rate schedulers modify the learning rate during training — common strategies include step decay, cosine annealing, and warmup — to balance initial progress with final precision.

Question 4

What is early stopping as a regularization technique?

Accepted Answer

Halting training when validation performance stops improving to prevent overfitting

Answer

Early stopping monitors validation loss and terminates training when it stops decreasing, saving the model checkpoint with the best generalization before overfitting occurs.

Question 5

What distinguishes overfitting from underfitting in a deep learning model?

Accepted Answer

Overfitting: low training error, high test error; underfitting: high training and test error

Answer

Overfitting occurs when a model memorizes training data, achieving low training error but poor generalization (high test error); underfitting occurs when the model is too simple to capture the data's patterns.

Deep Learning Practice Test

Deep Learning Practice Test

Deep Learning Training Techniques and Optimization