Neural Network Activation Functions and Optimization 2

Question 1

What is the GELU activation function and where is it commonly used?

Accepted Answer

Gaussian Error Linear Unit; commonly used in Transformers and BERT

Answer

Generalized Exponential Linear Unit; used in ResNets

Answer

Grouped Efficient Linear Unit; used in lightweight CNNs

Answer

Gated Exponential Learning Unit; used in LSTMs

Question 2

What is the difference between L1 and L2 regularization in neural networks?

Accepted Answer

L1 penalizes the sum of absolute weight values; L2 penalizes the sum of squared weight values

Answer

L1 applies to biases; L2 applies to weights only

Answer

L1 prevents overfitting; L2 prevents underfitting

Answer

L1 uses the first layer; L2 uses the second layer

Question 3

What is dropout regularization and how does it prevent overfitting?

Accepted Answer

It randomly deactivates a fraction of neurons during each training step, preventing co-adaptation

Answer

It removes neurons with small weights from the architecture permanently

Answer

It drops training samples with high loss values from each mini-batch

Answer

It reduces the learning rate whenever validation loss increases

Question 4

What is the swish activation function and why might it outperform ReLU?

Accepted Answer

f(x) = x · sigmoid(x), which is smooth and non-monotonic, often outperforming ReLU in deep networks

Answer

A piecewise linear function approximating sigmoid

Answer

A clipped version of tanh for better gradient flow

Answer

A learned activation function with trainable parameters

Question 5

What does the Adagrad optimizer do differently from standard SGD?

Accepted Answer

It adapts the learning rate per parameter based on the accumulated sum of squared gradients

Answer

It uses momentum from previous gradients

Answer

It uses second-order gradient information

Answer

It applies weight decay to all parameters uniformly

Question 6

What is the main weakness of Adagrad that RMSProp was designed to fix?

Accepted Answer

Adagrad's learning rate decreases monotonically and can become too small, stopping learning prematurely

Answer

Adagrad doesn't use per-parameter learning rates

Answer

Adagrad doesn't work for convex optimization

Answer

Adagrad requires computing the full Hessian matrix