CNN Batch Normalization and Dropout 3

Question 1

When both batch normalization and dropout are used in the same network, what ordering is generally recommended?

Accepted Answer

Conv → Batch Norm → ReLU → Dropout (in fully connected layers)

Answer

The standard practice is Conv → BN → ReLU for convolutional blocks, with dropout applied in fully connected layers after activation, as applying dropout before BN can disrupt the normalization statistics.

Question 2

How does layer normalization differ from batch normalization?

Accepted Answer

Layer normalization normalizes across the feature/channel dimension for each sample independently, not across the batch

Answer

Layer normalization normalizes across all features for a single training example rather than across examples in a batch, making it batch-size independent and effective for recurrent networks and transformers.

Question 3

What is group normalization and what problem does it solve compared to batch normalization?

Accepted Answer

It normalizes groups of channels together, working well even with batch size of 1

Answer

Group normalization divides channels into groups and normalizes within each group for each sample, making it effective for small batch sizes (like in object detection) where batch normalization degrades.

Question 4

Why can applying dropout before batch normalization cause problems?

Accepted Answer

Dropout changes the distribution of values BN sees, making its statistics inconsistent between training and inference

Answer

Dropout changes the variance of the layer's output during training; when dropout is removed at test time, BN's learned statistics no longer match the actual distribution, causing a variance shift problem.

Question 5

What is the effect of using a very high dropout rate (e.g., 0.9) on a CNN?

Accepted Answer

The network may underfit as too much information is lost during each forward pass

Answer

A very high dropout rate removes too many neurons per forward pass, causing the network to lose too much information and potentially underfit — regularization becomes so aggressive it prevents learning.

CNN - Convolutional Neural Networks Practice Test

CNN - Convolutional Neural Networks Practice Test

CNN Batch Normalization and Dropout 3