Neural Network Backpropagation and Training 3

Question 1

What is the exploding gradient problem and when is it most common?

Accepted Answer

Gradients grow exponentially, common in deep or recurrent networks

Answer

Exploding gradients occur when gradients grow exponentially during backpropagation, most commonly in very deep or recurrent networks with long sequences.

Question 2

What is second-order optimization and why is it rarely used in deep learning?

Accepted Answer

Using curvature information (Hessian) for updates, but too computationally expensive for large networks

Answer

Second-order methods use the Hessian matrix of second derivatives for more accurate updates, but computing and inverting the Hessian is computationally prohibitive for large networks.

Question 3

What does the 'dying ReLU' problem refer to?

Accepted Answer

Neurons that permanently output zero due to large negative biases

Answer

Dying ReLU occurs when neurons receive consistently negative inputs, causing them to always output zero and receive no gradient, effectively becoming permanently inactive.

Question 4

What is curriculum learning in neural network training?

Accepted Answer

Training the network on easier examples first, then gradually increasing difficulty

Answer

Curriculum learning organizes training data from easy to hard examples, mimicking how humans learn and often leading to better generalization and faster convergence.

Question 5

What is the role of the validation set during neural network training?

Accepted Answer

To tune hyperparameters and monitor for overfitting without contaminating the test set

Answer

The validation set provides an unbiased performance estimate during training for hyperparameter tuning and early stopping, keeping the test set truly held out.

Neural Network Practice Test

Neural Network Practice Test

Neural Network Backpropagation and Training 3