Deep Learning Recurrent Neural Networks 1 — Questions and Answers
Question 1: What distinguishes a recurrent neural network (RNN) from a feedforward network?
- RNNs have no activation functions
- RNNs have feedback connections that carry hidden state across time steps (Correct answer)
- RNNs only process fixed-size inputs
- RNNs use convolutional operations
Correct answer: RNNs have feedback connections that carry hidden state across time steps
RNNs pass a hidden state from one time step to the next, allowing the network to maintain memory of previous inputs in a sequence.
Question 2: What is the vanishing gradient problem particularly severe for in RNNs?
- Short sequences of 3–5 tokens
- Long-range dependencies spanning many time steps (Correct answer)
- Single-step predictions
- Batch normalization in recurrent layers
Correct answer: Long-range dependencies spanning many time steps
In RNNs, gradients must be backpropagated through many time steps, causing them to shrink exponentially and preventing the network from learning long-range dependencies.
Question 3: What are the key components of an LSTM cell that vanilla RNNs lack?
- Convolutional filters and pooling
- Forget gate, input gate, and output gate controlling information flow (Correct answer)
- Attention heads and positional encodings
- Batch normalization and dropout layers
Correct answer: Forget gate, input gate, and output gate controlling information flow
LSTMs introduce a cell state and three gating mechanisms — forget, input, and output — that regulate what information is retained, added, or read from memory.
Question 4: What problem does the forget gate in an LSTM address?
- Exploding activation values
- Deciding how much of the previous cell state to retain or discard (Correct answer)
- Mapping hidden states to output predictions
- Normalizing the input to the cell
Correct answer: Deciding how much of the previous cell state to retain or discard
The forget gate outputs values between 0 and 1 for each cell state element, controlling how much prior context is carried forward and what is discarded.
Question 5: What is backpropagation through time (BPTT)?
- Evaluating the model on future data
- Unrolling an RNN across time steps and applying backpropagation to compute gradients (Correct answer)
- Using time-series data to initialize RNN weights
- Applying dropout at each time step
Correct answer: Unrolling an RNN across time steps and applying backpropagation to compute gradients
BPTT unfolds the RNN computation graph across all time steps and applies the standard backpropagation algorithm to compute gradients with respect to all shared weights.
Question 6: What is a GRU (Gated Recurrent Unit) and how does it differ from an LSTM?
- A GRU has more gates than an LSTM
- A GRU simplifies LSTM by merging cell and hidden state and using only two gates (Correct answer)
- A GRU uses attention instead of recurrence
- A GRU replaces gradient-based learning with evolutionary methods
Correct answer: A GRU simplifies LSTM by merging cell and hidden state and using only two gates
GRUs combine the forget and input gates into a single update gate and merge the cell and hidden states, reducing parameters while achieving comparable performance to LSTMs on many tasks.
Deep Learning Recurrent Neural Networks 1
What distinguishes a recurrent neural network (RNN) from a feedforward network?