CNN Batch Normalization and Dropout 2 — Questions and Answers
Question 1: What does a dropout layer do during training?
- Randomly sets a fraction of neuron activations to zero (Correct answer)
- Randomly shuffles the order of filters in a convolutional layer
- Reduces the learning rate for randomly selected weights
- Clips gradient values above a threshold to zero
Correct answer: Randomly sets a fraction of neuron activations to zero
During training, dropout randomly sets a fraction of activations to zero with probability p (the dropout rate), forcing the network to learn redundant representations.
Question 2: What happens to dropout during model inference/testing?
- Dropout is applied with half the training dropout rate
- All neurons are active and outputs are scaled by the keep probability (Correct answer)
- Dropout continues but only in the final fully connected layer
- A new random mask is sampled and applied once at inference time
Correct answer: All neurons are active and outputs are scaled by the keep probability
At inference time, dropout is disabled and all neurons are active; outputs are typically scaled by the keep probability (1 - dropout rate) to match expected values during training.
Question 3: What is 'inverted dropout' and why is it preferred in practice?
- Scaling kept activations by 1/(1-p) during training so no scaling is needed at test time (Correct answer)
- Applying dropout only to the input layer and inverting the mask at deeper layers
- Using a dropout rate that increases with network depth
- Randomly scaling activations up instead of zeroing them down
Correct answer: Scaling kept activations by 1/(1-p) during training so no scaling is needed at test time
Inverted dropout scales kept activations by 1/(1-p) during training, so the network's expected output magnitude is unchanged and no scaling adjustment is needed at inference time.
Question 4: How does dropout act as a form of regularization?
- By penalizing large weight values via an L2 term in the loss function
- By forcing the network to learn robust features not dependent on specific co-adaptations (Correct answer)
- By reducing the number of parameters in the network permanently
- By increasing the effective learning rate to escape sharp minima
Correct answer: By forcing the network to learn robust features not dependent on specific co-adaptations
Dropout prevents co-adaptation of neurons by randomly removing them, forcing each neuron to learn useful features independently, which improves generalization.
Question 5: What is spatial dropout (also called 2D dropout) and when is it used?
- Dropping entire feature maps (channels) rather than individual activations in CNNs (Correct answer)
- Applying dropout only in the spatial height/width dimensions but keeping all channels
- Dropping activations in a structured grid pattern across the feature map
- Randomly masking contiguous rectangular regions of the input image
Correct answer: Dropping entire feature maps (channels) rather than individual activations in CNNs
Spatial dropout drops entire feature maps (channels) at once rather than individual elements, which is more effective for CNNs because adjacent pixels in a feature map are highly correlated.
Question 6: What dropout rate is commonly used as a starting default in fully connected layers of CNNs?
- 0.1 (drop 10% of neurons)
- 0.5 (drop 50% of neurons) (Correct answer)
- 0.9 (drop 90% of neurons)
- 0.01 (drop 1% of neurons)
Correct answer: 0.5 (drop 50% of neurons)
A dropout rate of 0.5 (dropping 50% of neurons) is a common default for fully connected layers, as suggested by the original dropout paper by Srivastava et al. (2014).
Question 7: Which interpretation explains why dropout works from an ensemble learning perspective?
- Each forward pass trains a different thinned network, and inference averages these networks (Correct answer)
- Dropout creates multiple copies of the network that are trained on different subsets of data
- Dropped neurons act as negative examples that improve contrastive learning
- Dropout forces the optimizer to use different learning rates for each layer
Correct answer: Each forward pass trains a different thinned network, and inference averages these networks
Each training step with dropout effectively trains a different sub-network; at test time, using all neurons approximates averaging over the exponentially many sub-networks, similar to an ensemble.
CNN Batch Normalization and Dropout 2
What does a dropout layer do during training?