Neural Network Recurrent Neural Networks 1 — Questions and Answers
Question 1: What makes recurrent neural networks (RNNs) different from feedforward networks?
- They use convolutional operations
- They maintain a hidden state that carries information across time steps (Correct answer)
- They do not use backpropagation
- They process inputs in random order
Correct answer: They maintain a hidden state that carries information across time steps
RNNs have recurrent connections that pass the hidden state from one time step to the next, allowing them to model sequential dependencies in variable-length inputs.
Question 2: What does LSTM stand for and what problem was it designed to solve?
- Long Short-Term Memory; vanishing/exploding gradients in standard RNNs (Correct answer)
- Large Scale Temporal Model; slow training in deep networks
- Layered Sequential Training Module; overfitting in time-series models
- Learned State Transition Memory; sparse gradient updates
Correct answer: Long Short-Term Memory; vanishing/exploding gradients in standard RNNs
LSTM (Long Short-Term Memory) was designed by Hochreiter and Schmidhuber to address vanishing and exploding gradients in standard RNNs by using gated cell states.
Question 3: What are the three gates in an LSTM cell?
- Input, process, output gates
- Forget, input, and output gates (Correct answer)
- Read, write, and erase gates
- Encode, decode, and attention gates
Correct answer: Forget, input, and output gates
An LSTM cell has forget gate (what to discard from cell state), input gate (what new information to add), and output gate (what to output as hidden state).
Question 4: What is a bidirectional RNN?
- An RNN that processes data both forwards and backwards in time simultaneously (Correct answer)
- An RNN that can handle both text and image inputs
- An RNN that uses two separate hidden states for long and short term
- An RNN where gradients flow in two directions during backpropagation
Correct answer: An RNN that processes data both forwards and backwards in time simultaneously
A bidirectional RNN processes sequences in both forward and backward temporal directions, concatenating or summing the two hidden states to capture context from both past and future.
Question 5: What is Backpropagation Through Time (BPTT)?
- A technique to speed up RNN training using time-based learning rates
- The algorithm for computing gradients in RNNs by unrolling the network across time steps (Correct answer)
- A method for transferring RNN weights across time periods
- A regularization technique specific to sequential models
Correct answer: The algorithm for computing gradients in RNNs by unrolling the network across time steps
BPTT unrolls the RNN across all time steps and applies standard backpropagation, computing gradients with respect to each time step's weights.
Question 6: What is the key difference between GRU and LSTM?
- GRU has no gating mechanism
- GRU has two gates instead of three and no separate cell state, making it simpler (Correct answer)
- GRU uses multiplicative interactions instead of additive
- GRU cannot handle long-term dependencies
Correct answer: GRU has two gates instead of three and no separate cell state, making it simpler
GRU (Gated Recurrent Unit) uses only reset and update gates (vs LSTM's three gates) and merges cell and hidden state, achieving similar performance with fewer parameters.
Neural Network Recurrent Neural Networks 1
What makes recurrent neural networks (RNNs) different from feedforward networks?