CNN Batch Normalization and Dropout 1 — Questions and Answers
Question 1: What does batch normalization normalize during training?
- The activations of each layer across a mini-batch (Correct answer)
- The weights of each convolutional filter
- The learning rate at each iteration
- The gradient magnitudes before backpropagation
Correct answer: The activations of each layer across a mini-batch
Batch normalization normalizes the activations (outputs) of a layer across the current mini-batch to have zero mean and unit variance.
Question 2: What are the two learnable parameters introduced by batch normalization?
- Scale (gamma) and shift (beta) (Correct answer)
- Mean (mu) and variance (sigma)
- Learning rate (alpha) and momentum (beta)
- Weight decay (lambda) and dropout rate (p)
Correct answer: Scale (gamma) and shift (beta)
Batch normalization introduces gamma (scale) and beta (shift) parameters that allow the network to undo the normalization if needed, learned during training.
Question 3: What problem does batch normalization primarily address?
- Vanishing gradients in shallow networks
- Internal covariate shift across layers (Correct answer)
- Overfitting due to excessive parameters
- Exploding activations in pooling layers
Correct answer: Internal covariate shift across layers
Batch normalization addresses internal covariate shift, which is the change in the distribution of layer inputs during training as preceding layer parameters change.
Question 4: During inference, how does batch normalization behave differently compared to training?
- It uses population statistics (running mean and variance) instead of batch statistics (Correct answer)
- It disables all normalization and passes activations unchanged
- It uses a larger batch to compute more accurate statistics
- It applies normalization only to the first and last layers
Correct answer: It uses population statistics (running mean and variance) instead of batch statistics
During inference, batch normalization uses fixed population statistics (running mean and variance accumulated during training) rather than computing batch statistics, ensuring consistent outputs for single samples.
Question 5: Where is batch normalization typically inserted in a convolutional block?
- After the activation function, before the pooling layer
- Before the activation function, after the convolutional layer (Correct answer)
- Before the convolutional layer, after the pooling layer
- Between the pooling layer and the fully connected layer only
Correct answer: Before the activation function, after the convolutional layer
Batch normalization is typically applied after the convolutional operation and before the activation function (e.g., Conv → BN → ReLU) to normalize pre-activation values.
Question 6: How does batch normalization generally affect the training speed of a CNN?
- It slows training by adding extra computation at each step
- It allows higher learning rates and accelerates convergence (Correct answer)
- It has no measurable effect on training speed
- It speeds up forward passes but significantly slows backpropagation
Correct answer: It allows higher learning rates and accelerates convergence
Batch normalization stabilizes gradient flow, reducing sensitivity to parameter initialization and allowing the use of higher learning rates, which accelerates convergence.
Question 7: What is a known limitation of batch normalization when using very small batch sizes?
- Learnable parameters gamma and beta become unstable
- Batch statistics become noisy and unreliable, degrading performance (Correct answer)
- The running mean is reset to zero after each batch
- It cannot be used with convolutional layers, only fully connected layers
Correct answer: Batch statistics become noisy and unreliable, degrading performance
With very small batch sizes (e.g., 1 or 2 samples), the per-batch mean and variance estimates are too noisy to be useful, making batch normalization less effective.
CNN Batch Normalization and Dropout 1
What does batch normalization normalize during training?