Neural Network Activation Functions and Optimization 1 β Questions and Answers
Question 1: What is the purpose of an activation function in a neural network?
- To normalize the inputs to each layer
- To introduce non-linearity so the network can learn complex patterns (Correct answer)
- To reduce the number of parameters in the model
- To compute the gradient during backpropagation
Correct answer: To introduce non-linearity so the network can learn complex patterns
Without activation functions, a neural network would be equivalent to a single linear transformation regardless of depth; activation functions introduce non-linearity enabling complex function approximation.
Question 2: What is the output range of the sigmoid activation function?
- (-β, +β)
- (-1, 1)
- (0, 1) (Correct answer)
- (-1, 0)
Correct answer: (0, 1)
The sigmoid function maps any real input to a value between 0 and 1, making it useful for binary classification output layers or gates in LSTMs.
Question 3: What distinguishes ReLU from sigmoid and tanh activation functions?
- ReLU is differentiable everywhere unlike sigmoid and tanh
- ReLU does not saturate for positive values, avoiding vanishing gradients in that region (Correct answer)
- ReLU outputs probabilities between 0 and 1
- ReLU is a second-order polynomial function
Correct answer: ReLU does not saturate for positive values, avoiding vanishing gradients in that region
ReLU (f(x) = max(0, x)) has a constant gradient of 1 for positive inputs, unlike sigmoid and tanh which saturate (near-zero gradients) at extreme values, alleviating vanishing gradients.
Question 4: What is the Leaky ReLU and how does it differ from standard ReLU?
- It applies a sigmoid function for negative inputs
- It allows a small non-zero gradient for negative inputs instead of outputting zero (Correct answer)
- It doubles the gradient for positive inputs
- It adds a learned bias to the ReLU output
Correct answer: It allows a small non-zero gradient for negative inputs instead of outputting zero
Leaky ReLU outputs a small slope (e.g., 0.01x) for negative inputs rather than zero, preventing neurons from dying and ensuring some gradient flow for negative activations.
Question 5: What is the softmax function used for in neural networks?
- To apply non-linear activation in hidden layers
- To convert raw logits into a probability distribution over classes (Correct answer)
- To normalize activations across spatial dimensions
- To reduce the output to a binary classification score
Correct answer: To convert raw logits into a probability distribution over classes
Softmax exponentiates and normalizes a vector of logits so they sum to 1, producing a valid probability distribution over mutually exclusive classes.
Question 6: What is the ELU (Exponential Linear Unit) activation and its advantage over ReLU?
- ELU outputs larger values for positive inputs, speeding up training
- ELU uses an exponential for negative inputs, producing negative mean activations that help self-normalize (Correct answer)
- ELU clips large activations to prevent gradient explosion
- ELU applies a learnable parameter to all inputs
Correct answer: ELU uses an exponential for negative inputs, producing negative mean activations that help self-normalize
ELU uses an exponential function for negative inputs, producing negative mean activations that push the mean closer to zero, reducing bias shift and often outperforming ReLU.
Neural Network Activation Functions and Optimization 1
What is the purpose of an activation function in a neural network?