DSE DSE - Data Science Deep Learning and Neural Networks 1 — Questions and Answers
Question 1: Which activation function is most commonly used in hidden layers of modern deep neural networks to mitigate the vanishing gradient problem?
- Sigmoid
- Tanh
- ReLU (Correct answer)
- Softmax
Correct answer: ReLU
ReLU (Rectified Linear Unit) outputs zero for negative inputs and the input value for positive inputs, providing sparse activations that help avoid vanishing gradients during training.
Question 2: What regularization technique prevents overfitting in neural networks by randomly setting a fraction of neuron activations to zero during each training step?
- L2 regularization
- Batch normalization
- Dropout (Correct answer)
- Early stopping
Correct answer: Dropout
Dropout randomly deactivates a proportion of neurons during each forward pass, forcing the network to learn redundant representations and reducing co-adaptation.
Question 3: Which optimization algorithm combines momentum with adaptive per-parameter learning rates and is widely used as a default optimizer for deep learning?
- SGD
- RMSProp
- Adam (Correct answer)
- Adagrad
Correct answer: Adam
Adam (Adaptive Moment Estimation) maintains both first-moment (momentum) and second-moment (per-parameter adaptive) estimates, making it robust and fast-converging for most deep learning tasks.
Question 4: What causes the 'vanishing gradient' problem in deep neural networks?
- Too many neurons per layer
- Gradients shrinking exponentially as they propagate backward through many layers with saturating activations (Correct answer)
- Using too large a learning rate
- Overfitting on the training set
Correct answer: Gradients shrinking exponentially as they propagate backward through many layers with saturating activations
When gradients are repeatedly multiplied by small values (from saturating activations like sigmoid) during backpropagation, they shrink exponentially, making early-layer weights nearly impossible to update.
Question 5: Which neural network architecture was specifically designed and is primarily used for image recognition and classification tasks?
- Recurrent Neural Network (RNN)
- Convolutional Neural Network (CNN) (Correct answer)
- Generative Adversarial Network (GAN)
- Transformer
Correct answer: Convolutional Neural Network (CNN)
CNNs use convolutional layers with shared weight filters to detect spatial hierarchies of features in images, making them the dominant architecture for computer vision tasks.
Question 6: In a neural network, what is the purpose of the backpropagation algorithm?
- Initializing weights before training
- Computing gradients of the loss with respect to each weight so they can be updated (Correct answer)
- Normalizing input features
- Selecting the optimal number of hidden layers
Correct answer: Computing gradients of the loss with respect to each weight so they can be updated
Backpropagation applies the chain rule to compute gradients of the loss function with respect to every weight in the network, enabling gradient-based optimization.
DSE DSE - Data Science Deep Learning and Neural Networks 1
Which activation function is most commonly used in hidden layers of modern deep neural networks to mitigate the vanishing gradient problem?