Neural Network Regularization and Overfitting 1 β Questions and Answers
Question 1: What does overfitting in a neural network mean?
- The model performs well on training data but poorly on unseen data (Correct answer)
- The model performs well on both training and test data
- The model fails to learn the training data patterns
- The model has too few parameters to fit the data
Correct answer: The model performs well on training data but poorly on unseen data
Overfitting occurs when a model learns the training data too well, including noise and irrelevant patterns, resulting in poor generalization to new data.
Question 2: Which regularization technique adds the sum of absolute values of weights to the loss function?
- L2 regularization
- L1 regularization (Correct answer)
- Dropout
- Batch normalization
Correct answer: L1 regularization
L1 regularization (Lasso) penalizes the loss function by adding the sum of the absolute values of the weights, promoting sparsity in the model.
Question 3: What is the primary effect of dropout regularization during training?
- It normalizes the activations at each layer
- It randomly deactivates a fraction of neurons, preventing co-adaptation (Correct answer)
- It reduces the learning rate over time
- It clips gradient values to prevent explosion
Correct answer: It randomly deactivates a fraction of neurons, preventing co-adaptation
Dropout randomly sets a fraction of neuron activations to zero during each training step, forcing the network to learn redundant representations and reducing co-adaptation.
Question 4: What is the bias-variance tradeoff in the context of neural networks?
- Balancing the number of layers against the number of neurons per layer
- Balancing underfitting (high bias) against overfitting (high variance) (Correct answer)
- Balancing training speed against model accuracy
- Balancing weight initialization against learning rate
Correct answer: Balancing underfitting (high bias) against overfitting (high variance)
The bias-variance tradeoff describes the tension between a model that is too simple (high bias, underfits) and one that is too complex (high variance, overfits).
Question 5: Which of the following is a symptom of underfitting in a neural network?
- High training accuracy and low test accuracy
- Low training accuracy and low test accuracy (Correct answer)
- High training accuracy and high test accuracy
- Low training accuracy and high test accuracy
Correct answer: Low training accuracy and low test accuracy
Underfitting occurs when the model is too simple to capture the underlying data patterns, resulting in poor performance on both the training set and the test set.
Question 6: What does L2 regularization (weight decay) do to the weights during training?
- It sets small weights to exactly zero
- It encourages weights to remain small by penalizing their squared magnitude (Correct answer)
- It increases large weights to improve learning speed
- It normalizes weights to have unit norm after each update
Correct answer: It encourages weights to remain small by penalizing their squared magnitude
L2 regularization adds the sum of squared weights to the loss, creating a penalty that shrinks weights toward zero proportionally, preventing any single weight from dominating.
Question 7: Early stopping is a regularization technique that halts training when:
- Training loss reaches zero
- Validation loss stops improving or begins to increase (Correct answer)
- The learning rate drops below a threshold
- All weight values converge to the same value
Correct answer: Validation loss stops improving or begins to increase
Early stopping monitors validation loss and stops training when it ceases to improve, preventing the model from continuing to overfit the training data.
Neural Network Regularization and Overfitting 1
What does overfitting in a neural network mean?