Training and Optimization Flashcards

A team is training a CNN for image classification on a dataset with limited samples. They notice the training accuracy is very high, but the validation accuracy is significantly lower and has plateaued. Which of the following techniques is specifically designed to address this issue by creating an ensemble of smaller networks during training?

Answer: Dropout

Dropout is a regularization technique that randomly sets a fraction of neuron activations to zero during each training step. [5] This prevents neurons from co-adapting too much and forces the network to learn more robust features, effectively training an ensemble of many smaller networks. [5, 7] This process helps to reduce overfitting, which is characterized by high training accuracy and poor validation/test accuracy. [22]

When applying transfer learning to a new image classification task with a very small and highly similar dataset compared to the original pre-trained model's dataset (e.g., ImageNet), what is the most appropriate strategy?

Answer: Freeze all convolutional layers and only train the final, newly added classification layers.

When the new dataset is small and similar to the original, the pre-trained convolutional layers already act as excellent generic feature extractors. [8] The best strategy is feature extraction: freeze the base model's layers to prevent overfitting on the small dataset and only train the new classifier head to adapt it to the new set of classes. [1, 23, 28] Re-training all layers (fine-tuning) would risk overfitting, while training from scratch would be ineffective with a small dataset. [23]

A CNN model's training loss is decreasing very slowly, and the model seems to be stuck. To accelerate convergence, the data scientist decides to switch from standard SGD to an adaptive learning rate optimizer. Which optimizer combines the benefits of both momentum and adaptive learning rates by keeping track of exponentially decaying averages of past gradients and past squared gradients?

Answer: Adam

The Adam optimizer, which stands for Adaptive Moment Estimation, computes adaptive learning rates for each parameter. It stores an exponentially decaying average of past squared gradients like RMSprop, and it also keeps an exponentially decaying average of past gradients, which is similar to momentum. [32, 35] This combination often leads to faster convergence than other optimizers like standard SGD or even RMSprop alone. [34]

During the training of a deep CNN, a practitioner observes that the distribution of inputs to deeper layers is constantly changing, a phenomenon known as internal covariate shift. This slows down training because the layers must continually adapt to a new distribution. Which technique is specifically designed to mitigate this problem by normalizing the inputs to each layer?

Answer: Batch Normalization

Batch Normalization is a technique designed to reduce internal covariate shift. [11] It normalizes the output of a previous activation layer by subtracting the batch mean and dividing by the batch standard deviation. [9, 11] This stabilization of the distributions of layer inputs allows for faster training, higher learning rates, and can act as a form of regularization. [10, 11]

A data scientist is training a CNN and wants to implement a learning rate schedule that starts with a relatively high learning rate and then smoothly decreases it following the shape of a cosine curve, potentially with periodic restarts. What is this scheduling strategy called?

Answer: Cosine Annealing

Cosine Annealing is a learning rate schedule where the learning rate is adjusted according to the cosine function. It starts with a higher value and smoothly 'anneals' or decreases to a minimum value. [26] This strategy can be very effective, especially when used with 'warm restarts,' where the learning rate is periodically reset to its initial high value, which can help the model escape poor local minima. [26]

Which of the following data augmentation techniques would be least appropriate for a CNN designed to classify handwritten digits from the MNIST dataset, where the orientation of the digit is a key feature?

Answer: Random vertical flipping

Data augmentation should create realistic variations of the training data while preserving the correct label. [29] For handwritten digits, a vertical flip would often change the digit into a different, non-existent, or incorrect digit (e.g., a '6' could look like a '9', but flipping a '6' vertically does not result in a valid '6'). Slight rotations, translations, and scaling are all plausible variations of handwriting and are therefore appropriate augmentation techniques. [4]