CNN Pooling and Activation Functions 5 — Questions and Answers
Question 1: What is the output range of the Tanh activation function?
- [0, 1]
- (-∞, +∞)
- [-1, 1] (Correct answer)
- [0, +∞)
Correct answer: [-1, 1]
Tanh maps all real inputs to the open interval (-1, 1), with outputs saturating near ±1 for large magnitude inputs.
Question 2: In a CNN used for multi-class image classification, which activation function is typically applied at the final output layer?
- ReLU
- Tanh
- Sigmoid
- Softmax (Correct answer)
Correct answer: Softmax
Softmax converts raw logits into a probability distribution over all classes, with each output representing the probability that the input belongs to that class.
Question 3: How does L2 pooling (also called square pooling) compute its output?
- Takes the element-wise maximum then applies L2 normalization
- Computes the square root of the mean of squared activations in the pooling region (Correct answer)
- Sums all activations and divides by the L2 norm of the kernel weights
- Applies L2 regularization to the activations before taking their average
Correct answer: Computes the square root of the mean of squared activations in the pooling region
L2 pooling computes sqrt(mean(x²)) over the pooling region, capturing energy rather than peak activation, and is related to the RMS of the region.
Question 4: Which statement correctly describes the GELU (Gaussian Error Linear Unit) activation function?
- GELU is a piecewise linear function with a fixed negative slope for x < 0
- GELU weights inputs by their Gaussian cumulative distribution, producing a smooth non-monotonic shape near zero (Correct answer)
- GELU clips all inputs to the range [-3, 3] using Gaussian thresholding
- GELU is identical to ReLU but with added Gaussian noise during training
Correct answer: GELU weights inputs by their Gaussian cumulative distribution, producing a smooth non-monotonic shape near zero
GELU multiplies the input by Φ(x), the standard Gaussian CDF, creating a smooth curve that gently gates activations and is used in models like BERT and GPT.
Question 5: Why might a practitioner choose strided convolutions over max pooling for downsampling in a modern CNN?
- Strided convolutions are parameter-free, reducing model size compared to pooling
- Strided convolutions allow the network to learn the optimal downsampling function rather than using a fixed rule (Correct answer)
- Strided convolutions always outperform max pooling on benchmark tasks regardless of architecture
- Strided convolutions prevent the vanishing gradient problem unlike max pooling
Correct answer: Strided convolutions allow the network to learn the optimal downsampling function rather than using a fixed rule
Strided convolutions have learnable weights, so the network can optimize how it reduces spatial resolution, whereas max pooling applies a fixed non-learned operation.
Question 6: What is the effect of using ReLU activation after a batch normalization layer versus before it?
- Applying BN before ReLU (BN→ReLU) is more common and typically avoids discarding normalized negative values immediately (Correct answer)
- Applying ReLU before BN (ReLU→BN) always leads to faster convergence in practice
- The order has no measurable effect since both operations are linear transformations
- Applying BN after ReLU doubles the gradient magnitude, improving deep network training
Correct answer: Applying BN before ReLU (BN→ReLU) is more common and typically avoids discarding normalized negative values immediately
The standard order is Conv→BN→ReLU; applying BN first normalizes the full distribution before ReLU discards negatives, preserving more information from normalization.
Question 7: Which scenario best illustrates the problem of 'activation saturation' in neural networks?
- A ReLU neuron outputting large positive values proportional to its input during forward pass
- A Sigmoid neuron receiving very large positive inputs, producing an output near 1 with near-zero gradient (Correct answer)
- A network with too few neurons, causing underfitting due to limited capacity
- A max pooling layer discarding too many activations, reducing feature map information
Correct answer: A Sigmoid neuron receiving very large positive inputs, producing an output near 1 with near-zero gradient
Saturation occurs when a Sigmoid (or Tanh) neuron's input is in the flat region of its curve, causing near-zero gradients that prevent effective weight updates during backpropagation.
CNN Pooling and Activation Functions 5
What is the output range of the Tanh activation function?