CNN - Convolutional Neural Networks Overfitting and Regularization Questions and Answers — Questions and Answers
Question 1: A CNN model achieves 99% accuracy on the training dataset but only 75% on the validation dataset. Which phenomenon is occurring, and what is a common technique to address it?
- Underfitting; increase model complexity by adding more layers.
- Overfitting; apply Dropout to the fully connected layers. (Correct answer)
- Data leakage; remove features that are present in both training and validation sets.
- Vanishing gradients; switch to a different activation function like Leaky ReLU.
Correct answer: Overfitting; apply Dropout to the fully connected layers.
The significant performance gap between training and validation accuracy is a classic sign of overfitting, where the model has learned the training data too well, including its noise, and fails to generalize to new data. [24, 27] Dropout is a widely used regularization technique that randomly sets a fraction of neuron activations to zero during training, which helps prevent complex co-adaptations and improves generalization. [3, 18, 20]
Question 2: What is the primary effect of L1 regularization on the weights of a CNN?
- It encourages all weights to become smaller and more evenly distributed.
- It has no direct effect on the weights but prunes entire network channels.
- It encourages some weights to become exactly zero, leading to a sparse model. (Correct answer)
- It normalizes the activations within each layer to have a mean of zero and a standard deviation of one.
Correct answer: It encourages some weights to become exactly zero, leading to a sparse model.
L1 regularization adds a penalty to the loss function proportional to the absolute value of the weights. [12] This has the effect of pushing the weights of less important features towards exactly zero, a process that results in a 'sparse' model where many weights are zero. [8, 9, 19] This can also be seen as a form of automatic feature selection.
Question 3: A data scientist is training a CNN on a small, specialized dataset of industrial parts. To prevent the model from memorizing the limited training examples, they decide to artificially expand the dataset. Which of the following techniques are they employing?
- L2 Regularization
- Early Stopping
- Principal Component Analysis (PCA)
- Data Augmentation (Correct answer)
Correct answer: Data Augmentation
Data augmentation is the process of artificially increasing the size of a training dataset by creating modified versions of the existing data. [15, 17] For images, this includes techniques like rotation, flipping, scaling, and color shifting. [4, 12, 18] This helps the model generalize better by exposing it to a wider variety of training examples, which is a powerful method for combating overfitting, especially with small datasets. [1, 15]
Question 4: Which of the following regularization techniques involves halting the training process when the model's performance on a validation set stops improving?
- Dropout
- Early Stopping (Correct answer)
- Weight Decay (L2)
- Batch Normalization
Correct answer: Early Stopping
Early stopping is a form of regularization where the model's performance on a separate validation dataset is monitored during training. If the validation performance (e.g., accuracy or loss) does not improve for a specified number of consecutive epochs (the 'patience'), the training process is halted to prevent the model from overfitting to the training data. [2, 5, 10, 16]
Question 5: What is the primary goal of using regularization techniques like Dropout or L2 regularization when training a Convolutional Neural Network?
- To increase the speed of convergence during training.
- To reduce the model's complexity and improve its ability to generalize to unseen data. (Correct answer)
- To increase the model's performance on the training dataset.
- To automatically determine the optimal number of convolutional layers.
Correct answer: To reduce the model's complexity and improve its ability to generalize to unseen data.
The fundamental purpose of regularization is to prevent overfitting. [2, 12] Overfitting occurs when a model learns the training data too well and fails to generalize to new, unseen data. [24, 27] Regularization techniques introduce constraints or penalties on the model's parameters (like L2 regularization) or its structure during training (like Dropout) to reduce its complexity and force it to learn more robust, generalizable patterns. [6, 11]
Question 6: Which of the following is NOT a common technique used to combat overfitting in a CNN?
- Increasing the number of epochs indefinitely. (Correct answer)
- Data Augmentation.
- Dropout.
- L2 Regularization (Weight Decay).
Correct answer: Increasing the number of epochs indefinitely.
Increasing the number of epochs indefinitely is likely to cause overfitting, not prevent it. As a model trains for longer, it has more opportunities to memorize the noise and specific details of the training data. [24] Data augmentation, Dropout, and L2 regularization are all standard and effective techniques used to reduce overfitting and improve a model's generalization capabilities. [12, 20]
CNN - Convolutional Neural Networks Overfitting and Regularization Questions and Answers
A CNN model achieves 99% accuracy on the training dataset but only 75% on the validation dataset.
Which phenomenon is occurring, and what is a common technique to address it?