Machine Learning Neural Networks 4 — Questions and Answers
Question 1: What is the exploding gradient problem and a common mitigation technique?
- Gradients becoming zero; solved by ReLU activation
- Gradients growing exponentially; mitigated by gradient clipping (Correct answer)
- Loss increasing unboundedly; fixed by dropout
- Weights diverging due to large batches; fixed by smaller learning rates
Correct answer: Gradients growing exponentially; mitigated by gradient clipping
Exploding gradients occur when backpropagated values grow exponentially; gradient clipping caps gradient norms to keep updates stable.
Question 2: In a CNN, what does 'pooling' accomplish?
- Adds learnable parameters to detect edges
- Reduces spatial dimensions while retaining dominant features (Correct answer)
- Concatenates feature maps from different layers
- Normalizes activations across the batch
Correct answer: Reduces spatial dimensions while retaining dominant features
Pooling (e.g., max pooling) downsamples feature maps by summarizing regions, reducing computation and providing spatial invariance.
Question 3: What is the difference between a shallow and a deep neural network?
- Shallow networks use more neurons per layer than deep ones
- Deep networks have more hidden layers than shallow networks (Correct answer)
- Shallow networks require more training data than deep ones
- Deep networks use simpler activation functions than shallow ones
Correct answer: Deep networks have more hidden layers than shallow networks
Depth refers to the number of hidden layers; deep networks have many hidden layers enabling hierarchical feature learning.
Question 4: What is the role of an embedding layer in a neural network processing text?
- It converts text to uppercase for normalization
- It maps discrete tokens to dense continuous vector representations (Correct answer)
- It applies convolutional filters across word sequences
- It computes attention scores between word pairs
Correct answer: It maps discrete tokens to dense continuous vector representations
An embedding layer learns a dense vector for each token, capturing semantic relationships in a continuous space.
Question 5: Which scenario best describes overfitting in a neural network?
- Training loss is high and validation loss is high
- Training loss is low but validation loss is much higher (Correct answer)
- Training loss decreases while validation loss also decreases steadily
- Gradients are near zero throughout training
Correct answer: Training loss is low but validation loss is much higher
Overfitting occurs when the model memorizes training data, leading to low training loss but poor generalization on unseen validation data.
Question 6: What is the purpose of a skip connection (residual connection) in ResNets?
- To skip certain training examples deemed too easy
- To allow gradients and inputs to bypass layers, easing training of very deep networks (Correct answer)
- To randomly deactivate entire layers during training
- To connect the first layer directly to the output layer
Correct answer: To allow gradients and inputs to bypass layers, easing training of very deep networks
Skip connections add the input of a block directly to its output, providing gradient shortcuts that alleviate vanishing gradients in very deep networks.
Question 7: In the context of neural networks, what does 'mini-batch gradient descent' mean?
- Computing gradients on the full dataset before each update
- Updating weights after computing gradients on a small subset of training data (Correct answer)
- Selecting only the hardest examples per batch
- Computing a separate gradient for each individual weight
Correct answer: Updating weights after computing gradients on a small subset of training data
Mini-batch gradient descent computes gradients over a small random subset (batch) of training examples, balancing efficiency and update noise.
Machine Learning Neural Networks 4
What is the exploding gradient problem and a common mitigation technique?