NLP Advanced Topics & Theory 3

Question 1

In seq2seq models, what problem does the attention mechanism primarily address?

Accepted Answer

The fixed-length bottleneck of the encoder's final hidden state

Answer

Attention lets the decoder dynamically focus on different encoder hidden states instead of compressing all source information into one vector.

Question 2

What distinguishes autoregressive language models (e.g., GPT) from masked language models (e.g., BERT)?

Accepted Answer

GPT predicts the next token left-to-right; BERT predicts masked tokens using full bidirectional context

Answer

GPT is a causal (unidirectional) model suited for generation, while BERT sees the entire context around each masked token.

Question 3

What is 'nucleus sampling' (top-p sampling) in text generation?

Accepted Answer

Sampling from the smallest set of tokens whose cumulative probability exceeds p

Answer

Top-p sampling accumulates tokens from highest to lowest probability until the cumulative mass reaches p, then samples from that dynamic set.

Question 4

Which metric evaluates machine translation quality by comparing n-gram overlap with reference translations?

Accepted Answer

BLEU

Answer

BLEU (Bilingual Evaluation Understudy) measures modified n-gram precision between hypothesis and reference translations with a brevity penalty.

Question 5

What is the 'exposure bias' problem in sequence-to-sequence training?

Accepted Answer

At test time the model sees its own predictions, but during training it always sees ground-truth tokens

Answer

Exposure bias arises because teacher-forcing at train time hides prediction errors, making the model fragile to its own mistakes at inference.

NLP Practice Test