Neural Network Backpropagation and Training 2 β Questions and Answers
Question 1: What is the Adam optimizer and what does it combine?
- A second-order optimizer using Hessians
- Adaptive learning rates using first and second moment estimates of gradients (Correct answer)
- A momentum-only optimizer with decay
- A stochastic optimizer with random restarts
Correct answer: Adaptive learning rates using first and second moment estimates of gradients
Adam combines adaptive learning rates (from RMSProp) with momentum by maintaining estimates of both the first moment (mean) and second moment (uncentered variance) of gradients.
Question 2: What is learning rate warmup in neural network training?
- Gradually increasing the learning rate at the start of training (Correct answer)
- Decreasing the learning rate at the end of training
- Using different learning rates for different layers
- Applying cyclic learning rate schedules
Correct answer: Gradually increasing the learning rate at the start of training
Learning rate warmup starts with a very small learning rate and gradually increases it during the early training steps to stabilize optimization.
Question 3: What is the purpose of weight initialization in neural networks?
- To set weights to zero for symmetry breaking
- To set initial weights appropriately to enable stable gradient flow from the start (Correct answer)
- To ensure all neurons produce the same initial output
- To minimize the initial loss value
Correct answer: To set initial weights appropriately to enable stable gradient flow from the start
Proper weight initialization ensures gradients neither vanish nor explode in early training, enabling efficient learning from the first iteration.
Question 4: What does the loss landscape of a neural network refer to?
- The visualization of training/validation loss curves
- The high-dimensional surface of the loss as a function of all weights (Correct answer)
- The plot of activations across layers
- The distribution of gradient magnitudes
Correct answer: The high-dimensional surface of the loss as a function of all weights
The loss landscape is the high-dimensional surface defined by the loss function over all possible weight configurations, containing minima, saddle points, and plateaus.
Question 5: What is early stopping as a regularization technique?
- Stopping training when gradients fall below a threshold
- Halting training when validation loss stops improving to prevent overfitting (Correct answer)
- Reducing the learning rate after a fixed number of epochs
- Removing neurons with small weights from the network
Correct answer: Halting training when validation loss stops improving to prevent overfitting
Early stopping monitors validation loss and halts training when it ceases to improve, preventing the model from overfitting the training data.
Question 6: In which scenario would you use gradient accumulation during training?
- When you want to use a very high learning rate
- When GPU memory is too small to fit large effective batch sizes (Correct answer)
- When the dataset has highly imbalanced classes
- When training with multiple output heads
Correct answer: When GPU memory is too small to fit large effective batch sizes
Gradient accumulation sums gradients over multiple small forward/backward passes before updating weights, simulating a larger batch size when memory is limited.
Neural Network Backpropagation and Training 2
What is the Adam optimizer and what does it combine?