Batch Normalization and Dropout Flashcards

What does batch normalization normalize during training?

Answer: The activations of each layer across a mini-batch

Batch normalization normalizes the activations (outputs) of a layer across the current mini-batch to have zero mean and unit variance.

What are the two learnable parameters introduced by batch normalization?

Answer: Scale (gamma) and shift (beta)

Batch normalization introduces gamma (scale) and beta (shift) parameters that allow the network to undo the normalization if needed, learned during training.

What problem does batch normalization primarily address?

Answer: Internal covariate shift across layers

Batch normalization addresses internal covariate shift, which is the change in the distribution of layer inputs during training as preceding layer parameters change.

During inference, how does batch normalization behave differently compared to training?

Answer: It uses population statistics (running mean and variance) instead of batch statistics

During inference, batch normalization uses fixed population statistics (running mean and variance accumulated during training) rather than computing batch statistics, ensuring consistent outputs for single samples.

Where is batch normalization typically inserted in a convolutional block?

Answer: Before the activation function, after the convolutional layer

Batch normalization is typically applied after the convolutional operation and before the activation function (e.g., Conv → BN → ReLU) to normalize pre-activation values.

How does batch normalization generally affect the training speed of a CNN?

Answer: It allows higher learning rates and accelerates convergence

Batch normalization stabilizes gradient flow, reducing sensitivity to parameter initialization and allowing the use of higher learning rates, which accelerates convergence.

What is a known limitation of batch normalization when using very small batch sizes?

Answer: Batch statistics become noisy and unreliable, degrading performance

With very small batch sizes (e.g., 1 or 2 samples), the per-batch mean and variance estimates are too noisy to be useful, making batch normalization less effective.