Overfitting and Regularization Flashcards

A CNN model achieves 99% accuracy on the training dataset but only 75% on the validation dataset. Which phenomenon is occurring, and what is a common technique to address it?

Answer: Overfitting; apply Dropout to the fully connected layers.

The significant performance gap between training and validation accuracy is a classic sign of overfitting, where the model has learned the training data too well, including its noise, and fails to generalize to new data. [24, 27] Dropout is a widely used regularization technique that randomly sets a fraction of neuron activations to zero during training, which helps prevent complex co-adaptations and improves generalization. [3, 18, 20]

What is the primary effect of L1 regularization on the weights of a CNN?

Answer: It encourages some weights to become exactly zero, leading to a sparse model.

L1 regularization adds a penalty to the loss function proportional to the absolute value of the weights. [12] This has the effect of pushing the weights of less important features towards exactly zero, a process that results in a 'sparse' model where many weights are zero. [8, 9, 19] This can also be seen as a form of automatic feature selection.

A data scientist is training a CNN on a small, specialized dataset of industrial parts. To prevent the model from memorizing the limited training examples, they decide to artificially expand the dataset. Which of the following techniques are they employing?

Answer: Data Augmentation

Data augmentation is the process of artificially increasing the size of a training dataset by creating modified versions of the existing data. [15, 17] For images, this includes techniques like rotation, flipping, scaling, and color shifting. [4, 12, 18] This helps the model generalize better by exposing it to a wider variety of training examples, which is a powerful method for combating overfitting, especially with small datasets. [1, 15]

Which of the following regularization techniques involves halting the training process when the model's performance on a validation set stops improving?

Answer: Early Stopping

Early stopping is a form of regularization where the model's performance on a separate validation dataset is monitored during training. If the validation performance (e.g., accuracy or loss) does not improve for a specified number of consecutive epochs (the 'patience'), the training process is halted to prevent the model from overfitting to the training data. [2, 5, 10, 16]

What is the primary goal of using regularization techniques like Dropout or L2 regularization when training a Convolutional Neural Network?

Answer: To reduce the model's complexity and improve its ability to generalize to unseen data.

The fundamental purpose of regularization is to prevent overfitting. [2, 12] Overfitting occurs when a model learns the training data too well and fails to generalize to new, unseen data. [24, 27] Regularization techniques introduce constraints or penalties on the model's parameters (like L2 regularization) or its structure during training (like Dropout) to reduce its complexity and force it to learn more robust, generalizable patterns. [6, 11]

Which of the following is NOT a common technique used to combat overfitting in a CNN?

Answer: Increasing the number of epochs indefinitely.

Increasing the number of epochs indefinitely is likely to cause overfitting, not prevent it. As a model trains for longer, it has more opportunities to memorize the noise and specific details of the training data. [24] Data augmentation, Dropout, and L2 regularization are all standard and effective techniques used to reduce overfitting and improve a model's generalization capabilities. [12, 20]