Deep Learning Training Techniques and Optimization

Question 1

What is the Adam optimizer and what are its two main components?

Accepted Answer

An optimizer combining adaptive learning rates (RMSProp) and momentum (first moment) with bias correction

Answer

A variant of SGD with only momentum

Answer

A second-order optimization method using the Hessian

Answer

An optimizer that applies gradient clipping automatically

Question 2

What is L2 regularization (weight decay) and how does it combat overfitting?

Accepted Answer

Adding a penalty proportional to the squared magnitude of weights to the loss, discouraging large weights

Answer

Adding a penalty proportional to the absolute value of weights to the loss

Answer

Dropping random neurons during training

Answer

Normalizing activations between layers

Question 3

What is the purpose of a learning rate scheduler during training?

Accepted Answer

Adjusting the learning rate over training time, typically decreasing it to enable fine-grained convergence

Answer

Setting the learning rate based on validation loss only

Answer

Choosing the optimizer algorithm dynamically

Answer

Scaling the learning rate by the number of layers

Question 4

What is early stopping as a regularization technique?

Accepted Answer

Halting training when validation performance stops improving to prevent overfitting

Answer

Stopping training when training loss reaches zero

Answer

Removing neurons after a fixed number of epochs

Answer

Reducing the learning rate after each epoch automatically

Question 5

What distinguishes overfitting from underfitting in a deep learning model?

Accepted Answer

Overfitting: low training error, high test error; underfitting: high training and test error

Answer

Overfitting: high training and test error; underfitting: low training error, high test error

Answer

Overfitting: model is too simple; underfitting: model is too complex

Answer

They describe the same phenomenon from different perspectives

Question 6

What is the purpose of a validation set in model training?

Accepted Answer

Providing an unbiased estimate of generalization during training for hyperparameter tuning

Answer

Training model weights by gradient descent

Answer

Serving as the final evaluation set reported to stakeholders

Answer

Augmenting the training set with additional examples