Neural Network Regularization and Overfitting 2 β Questions and Answers
Question 1: Which technique involves training multiple neural networks and averaging their predictions to reduce variance?
- Dropout
- Ensemble learning (Correct answer)
- Batch normalization
- Gradient clipping
Correct answer: Ensemble learning
Ensemble learning combines predictions from multiple independently trained models, which reduces variance and often improves generalization over any single model.
Question 2: Data augmentation helps reduce overfitting primarily by:
- Increasing the complexity of the model architecture
- Artificially expanding the training dataset with transformed samples (Correct answer)
- Removing noise from the training labels
- Reducing the number of trainable parameters
Correct answer: Artificially expanding the training dataset with transformed samples
Data augmentation creates additional training examples through transformations (flipping, rotation, cropping, etc.), effectively increasing dataset size and exposing the model to more variation.
Question 3: What is the typical dropout rate recommended for hidden layers in a deep neural network?
- 0.1 to 0.2
- 0.5 to 0.8
- 0.2 to 0.5 (Correct answer)
- 0.9 to 1.0
Correct answer: 0.2 to 0.5
A dropout rate of 0.2 to 0.5 (20β50% of neurons dropped) is commonly recommended for hidden layers, balancing regularization without excessively impeding learning.
Question 4: Which of the following best describes the concept of generalization in neural networks?
- The ability to train on large datasets efficiently
- The ability to perform well on new, unseen data beyond the training set (Correct answer)
- The ability to memorize all training examples accurately
- The ability to converge to a global minimum during optimization
Correct answer: The ability to perform well on new, unseen data beyond the training set
Generalization refers to a model's ability to apply learned patterns to new, previously unseen data rather than simply memorizing the training examples.
Question 5: In the context of regularization, what is 'weight decay'?
- Gradually reducing weights to zero after training completes
- An L2 penalty term added to the loss that shrinks weight magnitudes (Correct answer)
- Pruning weights below a threshold magnitude
- Normalizing weights to a fixed distribution each epoch
Correct answer: An L2 penalty term added to the loss that shrinks weight magnitudes
Weight decay is another name for L2 regularization, where a term proportional to the squared weight magnitudes is added to the loss function, discouraging large weights.
Question 6: Which of the following regularization strategies is most effective when the training dataset is very small?
- Increasing model depth
- Using dropout with a high drop rate of 0.9
- Data augmentation and transfer learning (Correct answer)
- Removing all regularization to preserve capacity
Correct answer: Data augmentation and transfer learning
When training data is scarce, data augmentation increases effective dataset size and transfer learning leverages pre-trained features, both significantly reducing overfitting.
Question 7: How does increasing the number of parameters in a neural network without sufficient data typically affect the bias-variance tradeoff?
- It increases bias and decreases variance
- It decreases bias but increases variance (Correct answer)
- It decreases both bias and variance equally
- It has no effect on the bias-variance tradeoff
Correct answer: It decreases bias but increases variance
More parameters give the model greater capacity to fit complex patterns (lower bias) but also make it more susceptible to fitting training noise (higher variance/overfitting).
Neural Network Regularization and Overfitting 2
Which technique involves training multiple neural networks and averaging their predictions to reduce variance?