CNN Batch Normalization and Dropout 3

Question 1

When both batch normalization and dropout are used in the same network, what ordering is generally recommended?

Accepted Answer

Conv → Batch Norm → ReLU → Dropout (in fully connected layers)

Answer

Conv → ReLU → Dropout → Batch Norm

Answer

Conv → Dropout → Batch Norm → ReLU

Answer

Batch Norm → Conv → Dropout → ReLU

Question 2

How does layer normalization differ from batch normalization?

Accepted Answer

Layer normalization normalizes across the feature/channel dimension for each sample independently, not across the batch

Answer

Layer normalization uses a larger batch size to compute more stable statistics

Answer

Layer normalization only normalizes the weight matrices, not the activations

Answer

Layer normalization applies normalization only to the final classification layer

Question 3

What is group normalization and what problem does it solve compared to batch normalization?

Accepted Answer

It normalizes groups of channels together, working well even with batch size of 1

Answer

It groups multiple batches together to compute more stable statistics

Answer

It normalizes activations only within a group of related convolutional layers

Answer

It groups parameters and applies shared normalization across filter groups

Question 4

Why can applying dropout before batch normalization cause problems?

Accepted Answer

Dropout changes the distribution of values BN sees, making its statistics inconsistent between training and inference

Answer

Dropout prevents the gamma and beta parameters of BN from being learned

Answer

Dropout zeros out the running mean, causing BN to fail at inference time

Answer

BN requires non-sparse inputs and dropout makes them too sparse to normalize correctly

Question 5

What is the effect of using a very high dropout rate (e.g., 0.9) on a CNN?

Accepted Answer

The network may underfit as too much information is lost during each forward pass

Answer

The network trains faster because fewer parameters need to be updated

Answer

The network becomes more accurate because more neurons are regularized

Answer

The batch normalization layers compensate automatically for the high dropout

Question 6

What is DropBlock, an extension of dropout designed specifically for CNNs?

Accepted Answer

A method that drops contiguous blocks of feature map regions rather than individual elements

Answer

A technique that drops entire convolutional blocks (Conv+BN+ReLU) during training

Answer

A regularization method that permanently removes underperforming filters after each epoch

Answer

A variant of dropout that drops activations based on their magnitude rather than randomly

Question 7

In the context of CNNs, what is the main advantage of using batch normalization over dropout for regularization in convolutional layers?

Accepted Answer

Batch normalization provides regularization while also stabilizing and accelerating training, without discarding information

Answer

Batch normalization uses fewer memory resources than dropout during training

Answer

Batch normalization can be applied to the input images, while dropout cannot

Answer

Batch normalization eliminates the need for weight initialization strategies