Neural Network Recurrent Neural Networks 2 β Questions and Answers
Question 1: What is a sequence-to-sequence (seq2seq) model and what tasks is it used for?
- A model that maps fixed-length inputs to fixed-length outputs
- An encoder-decoder RNN architecture that maps variable-length input sequences to variable-length output sequences (Correct answer)
- A model that processes sequences in parallel using attention
- A model that converts images to text using RNNs
Correct answer: An encoder-decoder RNN architecture that maps variable-length input sequences to variable-length output sequences
Seq2seq uses an encoder RNN to compress an input sequence into a context vector, and a decoder RNN to generate the output sequence, enabling tasks like machine translation.
Question 2: What problem does the attention mechanism solve in seq2seq models?
- It speeds up the encoder during training
- It allows the decoder to focus on relevant encoder states at each step instead of relying on a single context vector (Correct answer)
- It prevents the model from overfitting long sequences
- It enables parallel training of encoder and decoder
Correct answer: It allows the decoder to focus on relevant encoder states at each step instead of relying on a single context vector
Attention solves the bottleneck where all input information must pass through a single fixed-size context vector, by allowing the decoder to selectively attend to encoder hidden states.
Question 3: In a many-to-one RNN configuration, what type of task is being performed?
- Generating a sequence from a single input
- Classifying an entire sequence into a single output label (Correct answer)
- Transforming each input token to an output token
- Predicting multiple future time steps
Correct answer: Classifying an entire sequence into a single output label
Many-to-one RNNs process an entire input sequence and produce a single output, used in tasks like sentiment classification where a whole sentence produces one sentiment label.
Question 4: What is teacher forcing in RNN training?
- Using a teacher network to initialize student network weights
- Feeding ground-truth previous tokens as decoder inputs during training instead of the model's own predictions (Correct answer)
- Forcing the model to train for a minimum number of epochs
- A method for ensuring diverse training samples
Correct answer: Feeding ground-truth previous tokens as decoder inputs during training instead of the model's own predictions
Teacher forcing feeds the actual ground-truth token at each decoder step during training rather than the model's predicted token, accelerating convergence but potentially causing exposure bias.
Question 5: What is beam search in the context of RNN text generation?
- A gradient-based search for optimal RNN weights
- A decoding strategy that maintains the top-k most probable sequences at each step (Correct answer)
- A method for pruning RNN cells with low activation
- A technique for parallel decoding in recurrent models
Correct answer: A decoding strategy that maintains the top-k most probable sequences at each step
Beam search keeps the top-k (beam width) most probable partial sequences at each decoding step, trading off between greedy decoding and exhaustive search.
Question 6: What is the role of the hidden state in a vanilla RNN at each time step?
- It stores the complete sequence history losslessly
- It summarizes information from all previous time steps and is passed to the next step (Correct answer)
- It is reset to zero at each new time step
- It serves as the final output of the RNN
Correct answer: It summarizes information from all previous time steps and is passed to the next step
The hidden state in an RNN is a compressed summary of all previous inputs, computed as a function of the current input and previous hidden state, and passed forward to the next time step.
Neural Network Recurrent Neural Networks 2
What is a sequence-to-sequence (seq2seq) model and what tasks is it used for?