CNN - Convolutional Neural Networks Training and Optimization Questions and Answers — Questions and Answers
Question 1: A team is training a CNN for image classification on a dataset with limited samples. They notice the training accuracy is very high, but the validation accuracy is significantly lower and has plateaued. Which of the following techniques is specifically designed to address this issue by creating an ensemble of smaller networks during training?
- Batch Normalization
- Adam Optimizer
- Dropout (Correct answer)
- Data Augmentation
Correct answer: Dropout
Dropout is a regularization technique that randomly sets a fraction of neuron activations to zero during each training step. [5] This prevents neurons from co-adapting too much and forces the network to learn more robust features, effectively training an ensemble of many smaller networks. [5, 7] This process helps to reduce overfitting, which is characterized by high training accuracy and poor validation/test accuracy. [22]
Question 2: When applying transfer learning to a new image classification task with a very small and highly similar dataset compared to the original pre-trained model's dataset (e.g., ImageNet), what is the most appropriate strategy?
- Re-train the entire network from scratch with a very high learning rate.
- Freeze all convolutional layers and only train the final, newly added classification layers. (Correct answer)
- Unfreeze all layers and fine-tune the entire network with a moderately high learning rate.
- Use only the first few convolutional layers and build a new, deeper classifier on top.
Correct answer: Freeze all convolutional layers and only train the final, newly added classification layers.
When the new dataset is small and similar to the original, the pre-trained convolutional layers already act as excellent generic feature extractors. [8] The best strategy is feature extraction: freeze the base model's layers to prevent overfitting on the small dataset and only train the new classifier head to adapt it to the new set of classes. [1, 23, 28] Re-training all layers (fine-tuning) would risk overfitting, while training from scratch would be ineffective with a small dataset. [23]
Question 3: A CNN model's training loss is decreasing very slowly, and the model seems to be stuck. To accelerate convergence, the data scientist decides to switch from standard SGD to an adaptive learning rate optimizer. Which optimizer combines the benefits of both momentum and adaptive learning rates by keeping track of exponentially decaying averages of past gradients and past squared gradients?
- Adagrad
- Adam (Correct answer)
- Nesterov Accelerated Gradient
- RMSprop
Correct answer: Adam
The Adam optimizer, which stands for Adaptive Moment Estimation, computes adaptive learning rates for each parameter. It stores an exponentially decaying average of past squared gradients like RMSprop, and it also keeps an exponentially decaying average of past gradients, which is similar to momentum. [32, 35] This combination often leads to faster convergence than other optimizers like standard SGD or even RMSprop alone. [34]
Question 4: During the training of a deep CNN, a practitioner observes that the distribution of inputs to deeper layers is constantly changing, a phenomenon known as internal covariate shift. This slows down training because the layers must continually adapt to a new distribution. Which technique is specifically designed to mitigate this problem by normalizing the inputs to each layer?
- L2 Regularization
- Learning Rate Annealing
- Gradient Clipping
- Batch Normalization (Correct answer)
Correct answer: Batch Normalization
Batch Normalization is a technique designed to reduce internal covariate shift. [11] It normalizes the output of a previous activation layer by subtracting the batch mean and dividing by the batch standard deviation. [9, 11] This stabilization of the distributions of layer inputs allows for faster training, higher learning rates, and can act as a form of regularization. [10, 11]
Question 5: A data scientist is training a CNN and wants to implement a learning rate schedule that starts with a relatively high learning rate and then smoothly decreases it following the shape of a cosine curve, potentially with periodic restarts. What is this scheduling strategy called?
- Step Decay
- Exponential Decay
- Cosine Annealing (Correct answer)
- Time-Based Decay
Correct answer: Cosine Annealing
Cosine Annealing is a learning rate schedule where the learning rate is adjusted according to the cosine function. It starts with a higher value and smoothly 'anneals' or decreases to a minimum value. [26] This strategy can be very effective, especially when used with 'warm restarts,' where the learning rate is periodically reset to its initial high value, which can help the model escape poor local minima. [26]
Question 6: Which of the following data augmentation techniques would be least appropriate for a CNN designed to classify handwritten digits from the MNIST dataset, where the orientation of the digit is a key feature?
- Slight rotation (e.g., +/- 10 degrees)
- Random vertical flipping (Correct answer)
- Minor translation (shifting)
- Small scaling (zooming in/out)
Correct answer: Random vertical flipping
Data augmentation should create realistic variations of the training data while preserving the correct label. [29] For handwritten digits, a vertical flip would often change the digit into a different, non-existent, or incorrect digit (e.g., a '6' could look like a '9', but flipping a '6' vertically does not result in a valid '6'). Slight rotations, translations, and scaling are all plausible variations of handwriting and are therefore appropriate augmentation techniques. [4]
CNN - Convolutional Neural Networks Training and Optimization Questions and Answers
A team is training a CNN for image classification on a dataset with limited samples.
They notice the training accuracy is very high, but the validation accuracy is significantly lower and has plateaued.
Which of the following techniques is specifically designed to address this issue by creating an ensemble of smaller networks during training?