Neural Network Backpropagation and Training 2

Question 1

What is the Adam optimizer and what does it combine?

Accepted Answer

Adaptive learning rates using first and second moment estimates of gradients

Answer

Adam combines adaptive learning rates (from RMSProp) with momentum by maintaining estimates of both the first moment (mean) and second moment (uncentered variance) of gradients.

Question 2

What is learning rate warmup in neural network training?

Accepted Answer

Gradually increasing the learning rate at the start of training

Answer

Learning rate warmup starts with a very small learning rate and gradually increases it during the early training steps to stabilize optimization.

Question 3

What is the purpose of weight initialization in neural networks?

Accepted Answer

To set initial weights appropriately to enable stable gradient flow from the start

Answer

Proper weight initialization ensures gradients neither vanish nor explode in early training, enabling efficient learning from the first iteration.

Question 4

What does the loss landscape of a neural network refer to?

Accepted Answer

The high-dimensional surface of the loss as a function of all weights

Answer

The loss landscape is the high-dimensional surface defined by the loss function over all possible weight configurations, containing minima, saddle points, and plateaus.

Question 5

What is early stopping as a regularization technique?

Accepted Answer

Halting training when validation loss stops improving to prevent overfitting

Answer

Early stopping monitors validation loss and halts training when it ceases to improve, preventing the model from overfitting the training data.

Neural Network Practice Test

Neural Network Practice Test

Neural Network Backpropagation and Training 2