Deep Learning Neural Network Fundamentals 1 — Questions and Answers
Question 1: What is the primary purpose of an activation function in a neural network?
- To initialize weights
- To introduce non-linearity (Correct answer)
- To normalize inputs
- To reduce overfitting
Correct answer: To introduce non-linearity
Activation functions introduce non-linearity, enabling neural networks to learn complex patterns beyond simple linear mappings.
Question 2: Which activation function suffers from the 'dying ReLU' problem?
- Sigmoid
- Tanh
- ReLU (Correct answer)
- Softmax
Correct answer: ReLU
ReLU neurons can permanently output zero for all inputs when weights push the input to always be negative, rendering them inactive.
Question 3: What does the term 'vanishing gradient' describe in deep networks?
- Loss becoming too large
- Gradients shrinking near zero during backpropagation (Correct answer)
- Weights growing unboundedly
- Activations saturating at high values
Correct answer: Gradients shrinking near zero during backpropagation
Vanishing gradients occur when backpropagated error signals shrink exponentially through layers, preventing early layers from learning effectively.
Question 4: Which weight initialization method is specifically designed to work well with ReLU activations?
- Xavier initialization
- Zero initialization
- He initialization (Correct answer)
- Random uniform initialization
Correct answer: He initialization
He initialization sets initial weights with variance scaled by 2/n, accounting for the fact that ReLU zeroes out half its inputs.
Question 5: In a feedforward neural network, what direction does information flow during inference?
- Backward through layers
- Forward from input to output (Correct answer)
- Bidirectionally through all layers
- Only through skip connections
Correct answer: Forward from input to output
During inference, data passes forward from the input layer through hidden layers to the output layer without any backward pass.
Question 6: What is the role of bias terms in a neural network neuron?
- Scale the activation output
- Shift the activation function horizontally (Correct answer)
- Normalize the weight values
- Connect to skip connections
Correct answer: Shift the activation function horizontally
Bias terms allow the activation function to shift, enabling the network to fit patterns that don't pass through the origin.
Deep Learning Neural Network Fundamentals 1
What is the primary purpose of an activation function in a neural network?