Neural Network Activation Functions and Optimization

Question 1

What is the purpose of an activation function in a neural network?

Accepted Answer

To introduce non-linearity so the network can learn complex patterns

Answer

To normalize the inputs to each layer

Answer

To reduce the number of parameters in the model

Answer

To compute the gradient during backpropagation

Question 2

What is the output range of the sigmoid activation function?

Accepted Answer

(0, 1)

Answer

(-∞, +∞)

Answer

(-1, 1)

Answer

(-1, 0)

Question 3

What distinguishes ReLU from sigmoid and tanh activation functions?

Accepted Answer

ReLU does not saturate for positive values, avoiding vanishing gradients in that region

Answer

ReLU is differentiable everywhere unlike sigmoid and tanh

Answer

ReLU outputs probabilities between 0 and 1

Answer

ReLU is a second-order polynomial function

Question 4

What is the Leaky ReLU and how does it differ from standard ReLU?

Accepted Answer

It allows a small non-zero gradient for negative inputs instead of outputting zero

Answer

It applies a sigmoid function for negative inputs

Answer

It doubles the gradient for positive inputs

Answer

It adds a learned bias to the ReLU output

Question 5

What is the softmax function used for in neural networks?

Accepted Answer

To convert raw logits into a probability distribution over classes

Answer

To apply non-linear activation in hidden layers

Answer

To normalize activations across spatial dimensions

Answer

To reduce the output to a binary classification score

Question 6

What is the ELU (Exponential Linear Unit) activation and its advantage over ReLU?

Accepted Answer

ELU uses an exponential for negative inputs, producing negative mean activations that help self-normalize

Answer

ELU outputs larger values for positive inputs, speeding up training

Answer

ELU clips large activations to prevent gradient explosion

Answer

ELU applies a learnable parameter to all inputs