Neural Network Activation Functions and Optimization 2 — Questions and Answers
Question 1: What is the GELU activation function and where is it commonly used?
- Gaussian Error Linear Unit; commonly used in Transformers and BERT (Correct answer)
- Generalized Exponential Linear Unit; used in ResNets
- Grouped Efficient Linear Unit; used in lightweight CNNs
- Gated Exponential Learning Unit; used in LSTMs
Correct answer: Gaussian Error Linear Unit; commonly used in Transformers and BERT
GELU (Gaussian Error Linear Unit) weights inputs by their Gaussian CDF, providing a smooth, probabilistic gating effect; it's the default activation in BERT, GPT, and most modern Transformers.
Question 2: What is the difference between L1 and L2 regularization in neural networks?
- L1 penalizes the sum of absolute weight values; L2 penalizes the sum of squared weight values (Correct answer)
- L1 applies to biases; L2 applies to weights only
- L1 prevents overfitting; L2 prevents underfitting
- L1 uses the first layer; L2 uses the second layer
Correct answer: L1 penalizes the sum of absolute weight values; L2 penalizes the sum of squared weight values
L1 regularization (Lasso) adds the sum of absolute weight values to the loss, encouraging sparsity; L2 (Ridge/weight decay) adds squared weights, encouraging small but non-zero weights.
Question 3: What is dropout regularization and how does it prevent overfitting?
- It removes neurons with small weights from the architecture permanently
- It randomly deactivates a fraction of neurons during each training step, preventing co-adaptation (Correct answer)
- It drops training samples with high loss values from each mini-batch
- It reduces the learning rate whenever validation loss increases
Correct answer: It randomly deactivates a fraction of neurons during each training step, preventing co-adaptation
Dropout randomly sets a fraction of neuron activations to zero during each training forward pass, forcing the network to learn redundant representations and preventing co-adaptation.
Question 4: What is the swish activation function and why might it outperform ReLU?
- A piecewise linear function approximating sigmoid
- f(x) = x · sigmoid(x), which is smooth and non-monotonic, often outperforming ReLU in deep networks (Correct answer)
- A clipped version of tanh for better gradient flow
- A learned activation function with trainable parameters
Correct answer: f(x) = x · sigmoid(x), which is smooth and non-monotonic, often outperforming ReLU in deep networks
Swish (x·σ(x)) is smooth, non-monotonic, and unbounded above, properties that often lead to improved performance over ReLU in deeper architectures according to Google Brain research.
Question 5: What does the Adagrad optimizer do differently from standard SGD?
- It uses momentum from previous gradients
- It adapts the learning rate per parameter based on the accumulated sum of squared gradients (Correct answer)
- It uses second-order gradient information
- It applies weight decay to all parameters uniformly
Correct answer: It adapts the learning rate per parameter based on the accumulated sum of squared gradients
Adagrad adapts the learning rate for each parameter individually by dividing by the square root of accumulated squared gradients, giving smaller updates to frequently updated parameters.
Question 6: What is the main weakness of Adagrad that RMSProp was designed to fix?
- Adagrad doesn't use per-parameter learning rates
- Adagrad's learning rate decreases monotonically and can become too small, stopping learning prematurely (Correct answer)
- Adagrad doesn't work for convex optimization
- Adagrad requires computing the full Hessian matrix
Correct answer: Adagrad's learning rate decreases monotonically and can become too small, stopping learning prematurely
Adagrad's accumulation of all past squared gradients causes the learning rate to decrease monotonically to near-zero; RMSProp uses an exponential moving average to keep the rate from shrinking too much.
Neural Network Activation Functions and Optimization 2
What is the GELU activation function and where is it commonly used?