Neural Network Recurrent Neural Networks 3

Question 1

What is truncated backpropagation through time (TBPTT) and why is it used?

Accepted Answer

Limiting backpropagation to a fixed number of time steps to manage memory and computation

Answer

Stopping backpropagation when loss is below a threshold

Answer

Removing recurrent connections during inference

Answer

Applying dropout only at certain time steps

Question 2

What is a stateful RNN and when would you use it?

Accepted Answer

An RNN that preserves the hidden state between batches, useful for very long continuous sequences

Answer

An RNN that stores its weights in persistent storage

Answer

An RNN that uses attention to maintain state

Answer

An RNN that tracks the training loss state across epochs

Question 3

What is the peephole connection in LSTM variants?

Accepted Answer

Connections allowing the gates to observe the cell state directly when computing gate values

Answer

A shortcut connection from input to output bypassing the gates

Answer

A skip connection from early to late layers

Answer

An additional output gate for multi-task learning

Question 4

What is the primary weakness of standard RNNs compared to Transformers for long sequences?

Accepted Answer

RNNs process sequentially and struggle to capture very long-range dependencies due to gradient issues

Answer

RNNs cannot be trained with backpropagation

Answer

RNNs require more parameters than Transformers

Answer

RNNs cannot handle variable-length sequences

Question 5

What is a clockwork RNN?

Accepted Answer

An RNN with multiple groups of neurons that update at different temporal frequencies

Answer

An RNN that uses a clock signal to synchronize multiple GPUs

Answer

An RNN that processes inputs in fixed-size time windows

Answer

An RNN variant that uses periodic activation functions

Question 6

In sequence modeling, what does perplexity measure?

Accepted Answer

How well a probability distribution or language model predicts a sample sequence

Answer

The number of parameters in the recurrent model

Answer

The training time required for the RNN to converge

Answer

The ratio of correct to incorrect sequence predictions