CNN Training and Optimization 3 — Questions and Answers
Question 1: In transfer learning for CNNs, what is 'fine-tuning' as opposed to 'feature extraction'?
- Using only the final layer of a pretrained model
- Updating all or some pretrained weights on the new task (Correct answer)
- Freezing all pretrained weights during training
- Training only the classifier head
Correct answer: Updating all or some pretrained weights on the new task
Fine-tuning unfreezes some or all pretrained layers and continues training on the new dataset, allowing the model to adapt its representations to the target task.
Question 2: What is the vanishing gradient problem in deep CNNs and which activation function helps mitigate it?
- Gradients grow exponentially; sigmoid helps
- Gradients shrink toward zero in early layers; ReLU helps (Correct answer)
- Loss function becomes non-convex; tanh helps
- Weights oscillate during training; ELU helps
Correct answer: Gradients shrink toward zero in early layers; ReLU helps
Vanishing gradients occur when gradients diminish exponentially during backpropagation through many layers; ReLU avoids saturation for positive inputs, preserving gradient magnitude.
Question 3: What does the 'momentum' hyperparameter do in SGD with momentum for CNN training?
- Scales the learning rate adaptively per parameter
- Accumulates a fraction of past gradients to accelerate optimization (Correct answer)
- Adds L2 regularization to the loss
- Clips gradients to a maximum norm
Correct answer: Accumulates a fraction of past gradients to accelerate optimization
Momentum accumulates an exponentially decaying moving average of past gradients, helping accelerate SGD in relevant directions and dampening oscillations.
Question 4: Which CNN training technique uses multiple GPUs where each device holds a copy of the model and processes a different mini-batch?
- Model parallelism
- Data parallelism (Correct answer)
- Pipeline parallelism
- Tensor parallelism
Correct answer: Data parallelism
Data parallelism replicates the model on each GPU and splits mini-batches across devices, synchronizing gradients after each forward-backward pass.
Question 5: What is label smoothing in CNN training and what problem does it address?
- Augmenting labels with noise to prevent overfitting to overconfident predictions (Correct answer)
- Smoothing the loss curve with exponential moving average
- Randomly swapping class labels during training
- Applying softmax temperature scaling at test time
Correct answer: Augmenting labels with noise to prevent overfitting to overconfident predictions
Label smoothing replaces hard 0/1 targets with soft values (e.g., 0.9/0.1), preventing the model from becoming overconfident and improving generalization.
Question 6: In CNN optimization, what does 'weight decay' correspond to in terms of regularization?
- L1 regularization on weights
- L2 regularization on weights (Correct answer)
- Dropout on convolutional filters
- Elastic net regularization
Correct answer: L2 regularization on weights
Weight decay adds an L2 penalty on the magnitude of weights to the loss, which penalizes large weights and encourages the model to use smaller, more distributed weights.
Question 7: What is the 'dead ReLU' problem during CNN training?
- ReLU causes exploding gradients in deep networks
- Neurons with negative pre-activation permanently output zero and stop learning (Correct answer)
- ReLU activation leads to sparse but unreliable features
- ReLU causes the loss to become non-differentiable
Correct answer: Neurons with negative pre-activation permanently output zero and stop learning
Dead ReLUs occur when a neuron's input is always negative, causing it to output zero and receive zero gradient, effectively removing it from learning permanently.
CNN Training and Optimization 3
In transfer learning for CNNs, what is 'fine-tuning' as opposed to 'feature extraction'?