Deep Learning Recurrent Neural Networks 3

Question 1

What does 'stateful' mode mean for an RNN in frameworks like Keras?

Accepted Answer

The hidden state from the end of one batch is used as the initial state of the next batch

Answer

In stateful mode, the final hidden state of each batch becomes the initial hidden state for the subsequent batch, enabling processing of very long sequences across batches.

Question 2

What is the purpose of the output gate in an LSTM?

Accepted Answer

Controlling how much of the cell state is exposed as the hidden state output

Answer

The output gate determines what portion of the cell state is outputted as the hidden state, filtering which stored memory is relevant to the current prediction.

Question 3

How does an encoder-decoder architecture with attention differ from one without attention?

Accepted Answer

Attention allows the decoder to look at all encoder hidden states rather than only a single context vector

Answer

Attention mechanisms let the decoder compute a weighted combination of all encoder hidden states at each decoding step, overcoming the bottleneck of a single fixed context vector.

Question 4

What is the main limitation of vanilla Seq2Seq models for long input sequences?

Accepted Answer

The fixed-size context vector loses information when compressing long sequences

Answer

A single fixed-length context vector must encode all information from an arbitrarily long input, causing information loss for long sequences.

Question 5

What is the exploding gradient problem in RNNs?

Accepted Answer

Gradients growing exponentially through many time steps, causing unstable large weight updates

Answer

When gradients are multiplied across many time steps, they can grow exponentially, producing extremely large weight updates that destabilize or diverge training.

Deep Learning Practice Test

Deep Learning Practice Test

Deep Learning Recurrent Neural Networks 3