Neural Network Regularization and Overfitting 3

Question 1

What distinguishes L1 regularization from L2 regularization in terms of the weight solutions they produce?

Accepted Answer

L1 tends to produce sparse weights with some exactly zero; L2 produces small but non-zero weights

Answer

L1 produces dense small weights; L2 produces sparse weights

Answer

L1 and L2 produce identical weight distributions

Answer

L1 is used only for output layers; L2 is used only for hidden layers

Question 2

Which phenomenon occurs when a neural network's training loss is low but its generalization gap (train loss minus test loss) is large?

Accepted Answer

Overfitting

Answer

Underfitting

Answer

Saddle point convergence

Answer

Gradient vanishing

Question 3

Batch normalization reduces overfitting indirectly by:

Accepted Answer

Adding noise to the batch statistics, acting as a mild regularizer

Answer

Randomly zeroing out neuron outputs each step

Answer

Increasing the learning rate to escape local minima faster

Answer

Clipping gradient magnitudes to a maximum value

Question 4

What is the purpose of a validation set during neural network training?

Accepted Answer

To tune hyperparameters and monitor overfitting without contaminating test evaluation

Answer

To provide additional labeled examples that increase training set size

Answer

To compute the final accuracy reported in publications

Answer

To initialize weights before training begins

Question 5

Which of the following is NOT a commonly used regularization technique for neural networks?

Accepted Answer

Increasing the batch size indefinitely

Answer

Dropout

Answer

L2 weight decay

Answer

Early stopping

Question 6

What is the effect of noise injection as a regularization method in neural networks?

Accepted Answer

It improves robustness and generalization by making the model less sensitive to small input perturbations

Answer

It speeds up convergence by reducing the gradient magnitude

Answer

It removes outliers from the training batch automatically

Answer

It prevents gradient vanishing by adding values to activations

Question 7

In elastic net regularization, which combination of penalties is applied to the loss function?

Accepted Answer

A linear combination of both L1 and L2 penalties

Answer

L1 penalty only, applied to all layers

Answer

L2 penalty only, applied to output weights

Answer

A multiplicative combination of L1 and L2 penalties