Machine Learning Deep Learning 3 — Questions and Answers
Question 1: What distinguishes a recurrent neural network (RNN) from a standard feedforward network?
- RNNs use convolutional filters
- RNNs have connections that loop back, giving them memory of previous inputs (Correct answer)
- RNNs do not use backpropagation
- RNNs only work on image data
Correct answer: RNNs have connections that loop back, giving them memory of previous inputs
RNNs have recurrent connections that pass hidden state from one time step to the next, enabling them to model sequential data.
Question 2: What problem does an LSTM network solve compared to a vanilla RNN?
- It speeds up training with batch normalization
- It handles longer-term dependencies by using gating mechanisms (Correct answer)
- It removes the need for backpropagation
- It replaces activation functions with attention layers
Correct answer: It handles longer-term dependencies by using gating mechanisms
LSTM's cell state and gates (input, forget, output) allow it to retain or discard information over long sequences, mitigating the vanishing gradient problem in RNNs.
Question 3: In the transformer architecture, what does 'self-attention' compute?
- The gradient of the loss with respect to each token
- Weighted relationships between all positions in a sequence simultaneously (Correct answer)
- The number of attention heads needed for a task
- The positional encoding for each token
Correct answer: Weighted relationships between all positions in a sequence simultaneously
Self-attention computes a weighted sum of all token representations, where weights reflect pairwise relevance between tokens.
Question 4: Why is positional encoding added to token embeddings in a transformer?
- To reduce the embedding dimensionality
- To inject information about the order of tokens since transformers have no inherent sequential structure (Correct answer)
- To normalize the embeddings before attention
- To separate encoder and decoder embeddings
Correct answer: To inject information about the order of tokens since transformers have no inherent sequential structure
Transformers process all tokens in parallel and have no notion of order, so positional encodings provide sequence position information.
Question 5: What is weight initialization, and why does it matter in deep learning?
- Setting all weights to zero to start from a clean state
- Choosing initial weight values to prevent vanishing/exploding gradients and speed convergence (Correct answer)
- Freezing weights from a pretrained model
- Assigning weights based on the output class distribution
Correct answer: Choosing initial weight values to prevent vanishing/exploding gradients and speed convergence
Poor initialization (e.g., all zeros) causes symmetry problems; methods like Xavier or He initialization keep activations and gradients in a healthy range.
Question 6: What is the purpose of dropout regularization during training?
- To decrease the learning rate as training progresses
- To randomly deactivate neurons with probability p, reducing co-adaptation (Correct answer)
- To clip gradients to a maximum norm
- To remove neurons with the smallest weights
Correct answer: To randomly deactivate neurons with probability p, reducing co-adaptation
Dropout randomly zeroes neuron outputs during each forward pass, forcing the network to learn redundant representations and reducing overfitting.
Question 7: What is transfer learning in the context of deep neural networks?
- Moving a trained model from GPU to CPU
- Reusing a pretrained model's learned features as a starting point for a new task (Correct answer)
- Transferring data between training and validation sets
- Converting a model from one framework to another
Correct answer: Reusing a pretrained model's learned features as a starting point for a new task
Transfer learning leverages features learned on large datasets (e.g., ImageNet) and fine-tunes the model on a smaller target dataset, saving time and data.
Machine Learning Deep Learning 3
What distinguishes a recurrent neural network (RNN) from a standard feedforward network?