Neural Network Activation Functions and Optimization

Question 1

What is the purpose of an activation function in a neural network?

Accepted Answer

To introduce non-linearity so the network can learn complex patterns

Answer

Without activation functions, a neural network would be equivalent to a single linear transformation regardless of depth; activation functions introduce non-linearity enabling complex function approximation.

Question 2

What is the output range of the sigmoid activation function?

Accepted Answer

(0, 1)

Answer

The sigmoid function maps any real input to a value between 0 and 1, making it useful for binary classification output layers or gates in LSTMs.

Question 3

What distinguishes ReLU from sigmoid and tanh activation functions?

Accepted Answer

ReLU does not saturate for positive values, avoiding vanishing gradients in that region

Answer

ReLU (f(x) = max(0, x)) has a constant gradient of 1 for positive inputs, unlike sigmoid and tanh which saturate (near-zero gradients) at extreme values, alleviating vanishing gradients.

Question 4

What is the Leaky ReLU and how does it differ from standard ReLU?

Accepted Answer

It allows a small non-zero gradient for negative inputs instead of outputting zero

Answer

Leaky ReLU outputs a small slope (e.g., 0.01x) for negative inputs rather than zero, preventing neurons from dying and ensuring some gradient flow for negative activations.

Question 5

What is the softmax function used for in neural networks?

Accepted Answer

To convert raw logits into a probability distribution over classes

Answer

Softmax exponentiates and normalizes a vector of logits so they sum to 1, producing a valid probability distribution over mutually exclusive classes.

Neural Network Practice Test

Neural Network Practice Test

Neural Network Activation Functions and Optimization