CNN Pooling and Activation Functions 4 — Questions and Answers
Question 1: In Parametric ReLU (PReLU), what is learned during training?
- The threshold at which the function switches from linear to zero
- The slope of the activation for negative inputs, as a trainable parameter (Correct answer)
- The maximum activation value to prevent gradient explosion
- The learning rate applied specifically to ReLU neurons
Correct answer: The slope of the activation for negative inputs, as a trainable parameter
PReLU learns the negative-slope coefficient α via backpropagation, allowing the network to adapt the activation shape rather than fixing it at a preset value.
Question 2: Which pooling variant performs a weighted average where closer elements contribute more, based on their distance from the pooling region center?
- Stochastic pooling
- L2 pooling
- Spatial pyramid pooling
- Weighted average pooling (Correct answer)
Correct answer: Weighted average pooling
Weighted average pooling assigns higher weights to elements nearer the center of the pooling window, emphasizing spatially central activations.
Question 3: What is the key computational advantage of max pooling over learned downsampling (e.g., strided convolutions)?
- Max pooling introduces additional trainable weights that improve accuracy
- Max pooling has no trainable parameters, reducing model complexity and memory (Correct answer)
- Max pooling applies batch normalization implicitly during downsampling
- Max pooling increases feature map depth, capturing more information
Correct answer: Max pooling has no trainable parameters, reducing model complexity and memory
Max pooling is a fixed operation with no learnable parameters, making it cheaper in memory and compute compared to strided convolutions that add weights.
Question 4: SELU (Scaled Exponential Linear Unit) activation is designed to achieve what property automatically?
- Sparse activations with exactly half of neurons output zero
- Self-normalizing activations: mean near zero, variance near one across layers (Correct answer)
- Bounded outputs capped at ±1 to prevent gradient explosion
- Monotonically increasing outputs for all input values
Correct answer: Self-normalizing activations: mean near zero, variance near one across layers
SELU with its specific scale and slope parameters causes activations to converge toward zero mean and unit variance during forward propagation, enabling deep networks without batch norm.
Question 5: Spatial Pyramid Pooling (SPP) enables CNNs to accept inputs of varying sizes primarily by doing what?
- Resizing all inputs to a fixed dimension before the first convolution
- Applying pooling at multiple scales and concatenating outputs into a fixed-length vector (Correct answer)
- Using dilated convolutions to match arbitrary input resolutions
- Dynamically adjusting kernel sizes in convolutional layers based on input size
Correct answer: Applying pooling at multiple scales and concatenating outputs into a fixed-length vector
SPP pools the final feature maps at several granularities (e.g., 1×1, 2×2, 4×4) and concatenates the results, always producing a fixed-size representation regardless of input size.
Question 6: What happens to the gradient of the Sigmoid activation function when its input is very large (e.g., |x| > 5)?
- The gradient becomes very large, causing exploding gradients
- The gradient approaches zero, causing the vanishing gradient problem (Correct answer)
- The gradient remains constant at 0.25, the maximum for Sigmoid
- The gradient becomes negative, reversing weight update direction
Correct answer: The gradient approaches zero, causing the vanishing gradient problem
Sigmoid saturates near 0 and 1 for large |x|, making its derivative close to zero and causing gradients to vanish during backpropagation through multiple layers.
Question 7: In stochastic pooling, how is the pooled value selected during training?
- The maximum value in the pooling region, same as standard max pooling
- A random value sampled proportionally to the activation magnitudes in the region (Correct answer)
- The average of all activations after randomly dropping half of them
- A learned linear combination of all activations in the pooling window
Correct answer: A random value sampled proportionally to the activation magnitudes in the region
Stochastic pooling samples one activation from the pooling region with probability proportional to its magnitude, acting as a regularizer similar to dropout.
CNN Pooling and Activation Functions 4
In Parametric ReLU (PReLU), what is learned during training?