Deep Learning Recurrent Neural Networks 2 — Questions and Answers
Question 1: What is a bidirectional RNN?
- An RNN that processes images from multiple angles
- An RNN that processes the sequence in both forward and backward directions simultaneously (Correct answer)
- An RNN with two hidden layers stacked vertically
- An RNN that uses two different loss functions
Correct answer: An RNN that processes the sequence in both forward and backward directions simultaneously
A bidirectional RNN runs two RNNs on the input — one forward and one backward — and concatenates their outputs, giving each step access to both past and future context.
Question 2: In sequence-to-sequence (Seq2Seq) models, what is the role of the encoder?
- Generating the output sequence token by token
- Compressing the input sequence into a fixed-length context vector (Correct answer)
- Applying attention over the input
- Computing cross-entropy loss over predictions
Correct answer: Compressing the input sequence into a fixed-length context vector
The encoder processes the input sequence and compresses it into a context vector (the final hidden state) that summarizes the input for the decoder.
Question 3: What is teacher forcing in RNN training?
- Using a separate teacher network to guide training
- Feeding the ground-truth previous output as the next input during training rather than the model's own prediction (Correct answer)
- Clipping gradients during backpropagation
- Initializing weights from a pretrained language model
Correct answer: Feeding the ground-truth previous output as the next input during training rather than the model's own prediction
Teacher forcing uses the ground-truth token at each step as the next input during training, stabilizing learning by avoiding compounding errors from the model's own predictions.
Question 4: What is gradient clipping used for when training RNNs?
- Preventing the loss from going below zero
- Capping gradient magnitudes to prevent exploding gradients from destabilizing training (Correct answer)
- Increasing the learning rate dynamically
- Removing neurons with zero gradients
Correct answer: Capping gradient magnitudes to prevent exploding gradients from destabilizing training
Gradient clipping rescales gradients when their norm exceeds a threshold, preventing the large weight updates that cause divergence due to exploding gradients in deep or long RNNs.
Question 5: What type of tasks are RNNs / LSTMs most naturally suited for?
- Static image classification
- Sequential and time-series tasks such as language modeling, speech recognition, and forecasting (Correct answer)
- Graph-structured data
- Clustering unlabeled datasets
Correct answer: Sequential and time-series tasks such as language modeling, speech recognition, and forecasting
RNNs and LSTMs are designed for sequential data where order matters, making them well-suited for text, audio, time series, and other temporal tasks.
Question 6: What is the hidden state in an RNN?
- The final layer's output probabilities
- A vector summarizing information from previous time steps passed to the next step (Correct answer)
- The loss value at each step
- The weight matrix connecting input to hidden layer
Correct answer: A vector summarizing information from previous time steps passed to the next step
The hidden state is a learned vector representation that encodes relevant information from all previous time steps and is updated at each new input.
Deep Learning Recurrent Neural Networks 2
What is a bidirectional RNN?