Neural Network Recurrent Neural Networks 3

Question 1

What is truncated backpropagation through time (TBPTT) and why is it used?

Accepted Answer

Limiting backpropagation to a fixed number of time steps to manage memory and computation

Answer

TBPTT divides long sequences into segments and backpropagates gradients only a fixed number of steps back, making training of very long sequences tractable.

Question 2

What is a stateful RNN and when would you use it?

Accepted Answer

An RNN that preserves the hidden state between batches, useful for very long continuous sequences

Answer

A stateful RNN retains the final hidden state of one batch as the initial state of the next, enabling processing of sequences longer than the batch window across consecutive batches.

Question 3

What is the peephole connection in LSTM variants?

Accepted Answer

Connections allowing the gates to observe the cell state directly when computing gate values

Answer

Peephole connections allow the forget, input, and output gates to look at the current cell state (not just the hidden state), giving gates access to more information for their decisions.

Question 4

What is the primary weakness of standard RNNs compared to Transformers for long sequences?

Accepted Answer

RNNs process sequentially and struggle to capture very long-range dependencies due to gradient issues

Answer

RNNs process tokens sequentially and pass information through the hidden state, making it hard to preserve information from very distant time steps due to vanishing gradients.

Question 5

What is a clockwork RNN?

Accepted Answer

An RNN with multiple groups of neurons that update at different temporal frequencies

Answer

A clockwork RNN divides hidden neurons into groups that operate at different time scales (frequencies), allowing faster groups to handle short-term and slower groups to handle long-term patterns.

Neural Network Practice Test

Neural Network Practice Test

Neural Network Recurrent Neural Networks 3