Neural Network Backpropagation and Training 1 — Questions and Answers
Question 1: What is the vanishing gradient problem in deep neural networks?
- Gradients become too large during backpropagation
- Gradients become extremely small and fail to update early layers (Correct answer)
- The loss function produces zero gradients
- Weight updates cause NaN values
Correct answer: Gradients become extremely small and fail to update early layers
The vanishing gradient problem occurs when gradients shrink exponentially as they propagate back through many layers, leaving early layers barely updated.
Question 2: Which technique clips gradient magnitudes to prevent exploding gradients during training?
- Dropout
- Gradient clipping (Correct answer)
- Weight decay
- Batch normalization
Correct answer: Gradient clipping
Gradient clipping caps the gradient norm or value at a maximum threshold, preventing exploding gradients that destabilize training.
Question 3: In mini-batch gradient descent, what happens to the gradient estimate as batch size increases?
- It becomes noisier
- It becomes a more accurate estimate of the true gradient (Correct answer)
- It always leads to faster convergence
- It eliminates the need for learning rate tuning
Correct answer: It becomes a more accurate estimate of the true gradient
Larger batch sizes average gradients over more samples, producing a less noisy, more accurate estimate of the true population gradient.
Question 4: What does the learning rate control in neural network training?
- The number of training epochs
- The step size taken in the direction of the negative gradient (Correct answer)
- The proportion of neurons dropped out
- The ratio of training to validation data
Correct answer: The step size taken in the direction of the negative gradient
The learning rate determines how large a step the optimizer takes in the direction of the negative gradient during each weight update.
Question 5: What is the purpose of momentum in gradient descent optimization?
- To reduce the learning rate over time
- To accumulate past gradients to accelerate and smooth updates (Correct answer)
- To randomly perturb weights during training
- To normalize gradients before applying updates
Correct answer: To accumulate past gradients to accelerate and smooth updates
Momentum accumulates a velocity vector from past gradients, helping the optimizer accelerate in consistent directions and dampen oscillations.
Question 6: What does the chain rule of calculus enable in backpropagation?
- Parallel computation across layers
- Computing gradients through composite functions layer by layer (Correct answer)
- Storing activations for forward pass reuse
- Regularizing weight magnitudes automatically
Correct answer: Computing gradients through composite functions layer by layer
The chain rule allows backpropagation to compute gradients of the loss with respect to each weight by multiplying local gradients through the network.
Neural Network Backpropagation and Training 1
What is the vanishing gradient problem in deep neural networks?