Natural Language Processing Language Models and Neural NLP 2

Question 1

What is transfer learning in the context of NLP?

Accepted Answer

Pre-training a large model on general text data then fine-tuning it on a specific downstream task

Answer

Transfer learning allows NLP models to leverage knowledge from large-scale pre-training and adapt it to task-specific data with minimal additional training.

Question 2

What is the vanishing gradient problem in RNNs?

Accepted Answer

Gradients become extremely small during backpropagation through long sequences, preventing the model from learning long-range dependencies

Answer

In deep or long-sequence RNNs, gradients shrink exponentially as they propagate back through time steps, making it hard to learn dependencies between distant tokens.

Question 3

What gates does an LSTM unit use to control information flow?

Accepted Answer

Input gate, forget gate, and output gate

Answer

LSTMs use three gates—input (what new info to store), forget (what to discard from cell state), and output (what to expose as the hidden state)—to regulate long-term memory.

Question 4

What is positional encoding used for in transformer models?

Accepted Answer

To inject information about each token's position in the sequence since self-attention is permutation-invariant

Answer

Because self-attention treats the sequence as a set, positional encodings add order information so the model knows which token comes first, second, etc.

Question 5

What is perplexity as a metric for language models?

Accepted Answer

The exponentiated average negative log-likelihood of a test corpus; lower perplexity means better prediction

Answer

Perplexity measures how surprised a language model is by unseen text — a perplexity of N means the model is as confused as if it had to choose uniformly among N options at each step.

Natural Language Processing Practice Test

Natural Language Processing Practice Test

Natural Language Processing Language Models and Neural NLP 2