Neural Network Backpropagation and Training

Question 1

What is the vanishing gradient problem in deep neural networks?

Accepted Answer

Gradients become extremely small and fail to update early layers

Answer

The vanishing gradient problem occurs when gradients shrink exponentially as they propagate back through many layers, leaving early layers barely updated.

Question 2

Which technique clips gradient magnitudes to prevent exploding gradients during training?

Accepted Answer

Gradient clipping

Answer

Gradient clipping caps the gradient norm or value at a maximum threshold, preventing exploding gradients that destabilize training.

Question 3

In mini-batch gradient descent, what happens to the gradient estimate as batch size increases?

Accepted Answer

It becomes a more accurate estimate of the true gradient

Answer

Larger batch sizes average gradients over more samples, producing a less noisy, more accurate estimate of the true population gradient.

Question 4

What does the learning rate control in neural network training?

Accepted Answer

The step size taken in the direction of the negative gradient

Answer

The learning rate determines how large a step the optimizer takes in the direction of the negative gradient during each weight update.

Question 5

What is the purpose of momentum in gradient descent optimization?

Accepted Answer

To accumulate past gradients to accelerate and smooth updates

Answer

Momentum accumulates a velocity vector from past gradients, helping the optimizer accelerate in consistent directions and dampen oscillations.

Neural Network Practice Test

Neural Network Practice Test

Neural Network Backpropagation and Training