Neural Network Regularization and Overfitting 3

Question 1

What distinguishes L1 regularization from L2 regularization in terms of the weight solutions they produce?

Accepted Answer

L1 tends to produce sparse weights with some exactly zero; L2 produces small but non-zero weights

Answer

L1 regularization's absolute value penalty creates a sparse solution by driving some weights to exactly zero, while L2's squared penalty shrinks all weights toward zero but rarely makes them exactly zero.

Question 2

Which phenomenon occurs when a neural network's training loss is low but its generalization gap (train loss minus test loss) is large?

Accepted Answer

Overfitting

Answer

A large gap between low training loss and high test loss is the classic signature of overfitting — the model has memorized training data rather than learning generalizable patterns.

Question 3

Batch normalization reduces overfitting indirectly by:

Accepted Answer

Adding noise to the batch statistics, acting as a mild regularizer

Answer

Batch normalization normalizes activations using mini-batch statistics, introducing slight noise in the estimates that acts as a mild regularizer and often reduces the need for dropout.

Question 4

What is the purpose of a validation set during neural network training?

Accepted Answer

To tune hyperparameters and monitor overfitting without contaminating test evaluation

Answer

The validation set is used during training to evaluate hyperparameter choices and detect overfitting, while the test set remains held out for final unbiased evaluation.

Question 5

Which of the following is NOT a commonly used regularization technique for neural networks?

Accepted Answer

Increasing the batch size indefinitely

Answer

Indefinitely increasing batch size is not a regularization technique; in fact, very large batches can hurt generalization by producing sharp minima.

Neural Network Practice Test

Neural Network Practice Test

Neural Network Regularization and Overfitting 3