Neural Network Activation Functions and Optimization 2

Question 1

What is the GELU activation function and where is it commonly used?

Accepted Answer

Gaussian Error Linear Unit; commonly used in Transformers and BERT

Answer

GELU (Gaussian Error Linear Unit) weights inputs by their Gaussian CDF, providing a smooth, probabilistic gating effect; it's the default activation in BERT, GPT, and most modern Transformers.

Question 2

What is the difference between L1 and L2 regularization in neural networks?

Accepted Answer

L1 penalizes the sum of absolute weight values; L2 penalizes the sum of squared weight values

Answer

L1 regularization (Lasso) adds the sum of absolute weight values to the loss, encouraging sparsity; L2 (Ridge/weight decay) adds squared weights, encouraging small but non-zero weights.

Question 3

What is dropout regularization and how does it prevent overfitting?

Accepted Answer

It randomly deactivates a fraction of neurons during each training step, preventing co-adaptation

Answer

Dropout randomly sets a fraction of neuron activations to zero during each training forward pass, forcing the network to learn redundant representations and preventing co-adaptation.

Question 4

What is the swish activation function and why might it outperform ReLU?

Accepted Answer

f(x) = x · sigmoid(x), which is smooth and non-monotonic, often outperforming ReLU in deep networks

Answer

Swish (x·σ(x)) is smooth, non-monotonic, and unbounded above, properties that often lead to improved performance over ReLU in deeper architectures according to Google Brain research.

Question 5

What does the Adagrad optimizer do differently from standard SGD?

Accepted Answer

It adapts the learning rate per parameter based on the accumulated sum of squared gradients

Answer

Adagrad adapts the learning rate for each parameter individually by dividing by the square root of accumulated squared gradients, giving smaller updates to frequently updated parameters.

Neural Network Practice Test

Neural Network Practice Test

Neural Network Activation Functions and Optimization 2