Neural Network Backpropagation and Training 3

Question 1

What is the exploding gradient problem and when is it most common?

Accepted Answer

Gradients grow exponentially, common in deep or recurrent networks

Answer

Gradients become zero, common in shallow networks

Answer

Gradients oscillate, common in CNNs

Answer

Gradients become NaN only at the output layer

Question 2

What is second-order optimization and why is it rarely used in deep learning?

Accepted Answer

Using curvature information (Hessian) for updates, but too computationally expensive for large networks

Answer

Optimization using two loss functions simultaneously

Answer

Training with two optimizers in parallel

Answer

Using second derivatives only for the output layer

Question 3

What does the 'dying ReLU' problem refer to?

Accepted Answer

Neurons that permanently output zero due to large negative biases

Answer

ReLU activations that oscillate between 0 and 1

Answer

ReLU gradients that become too large

Answer

ReLU units that saturate at very high values

Question 4

What is curriculum learning in neural network training?

Accepted Answer

Training the network on easier examples first, then gradually increasing difficulty

Answer

Presenting training samples in random order

Answer

Using a fixed curriculum of hyperparameter schedules

Answer

Applying different learning rates to different layers

Question 5

What is the role of the validation set during neural network training?

Accepted Answer

To tune hyperparameters and monitor for overfitting without contaminating the test set

Answer

To compute the final test accuracy of the model

Answer

To augment the training data distribution

Answer

To initialize the model weights

Question 6

What does 'label smoothing' do to the target distribution in classification?

Accepted Answer

It replaces hard one-hot labels with a soft distribution that assigns small probability to other classes

Answer

It replaces all labels with uniform probabilities

Answer

It randomly flips labels during training

Answer

It normalizes labels to have zero mean