NLP Advanced Topics & Theory 3 — Questions and Answers
Question 1: In seq2seq models, what problem does the attention mechanism primarily address?
- Slow training speed on long sequences
- The fixed-length bottleneck of the encoder's final hidden state (Correct answer)
- Out-of-vocabulary token handling
- Gradient vanishing in the decoder LSTM
Correct answer: The fixed-length bottleneck of the encoder's final hidden state
Attention lets the decoder dynamically focus on different encoder hidden states instead of compressing all source information into one vector.
Question 2: What distinguishes autoregressive language models (e.g., GPT) from masked language models (e.g., BERT)?
- GPT uses bidirectional context; BERT uses left-to-right context only
- GPT predicts the next token left-to-right; BERT predicts masked tokens using full bidirectional context (Correct answer)
- GPT requires fine-tuning; BERT generates text without fine-tuning
- GPT uses absolute positional encoding; BERT uses relative positional encoding
Correct answer: GPT predicts the next token left-to-right; BERT predicts masked tokens using full bidirectional context
GPT is a causal (unidirectional) model suited for generation, while BERT sees the entire context around each masked token.
Question 3: What is 'nucleus sampling' (top-p sampling) in text generation?
- Sampling from the top-k tokens sorted by logit magnitude
- Sampling from the smallest set of tokens whose cumulative probability exceeds p (Correct answer)
- Scaling the temperature parameter to p before sampling
- Sampling from tokens with probability exactly equal to p
Correct answer: Sampling from the smallest set of tokens whose cumulative probability exceeds p
Top-p sampling accumulates tokens from highest to lowest probability until the cumulative mass reaches p, then samples from that dynamic set.
Question 4: Which metric evaluates machine translation quality by comparing n-gram overlap with reference translations?
- ROUGE
- METEOR
- BLEU (Correct answer)
- BERTScore
Correct answer: BLEU
BLEU (Bilingual Evaluation Understudy) measures modified n-gram precision between hypothesis and reference translations with a brevity penalty.
Question 5: What is the 'exposure bias' problem in sequence-to-sequence training?
- The model is exposed to too many training examples, causing overfitting
- At test time the model sees its own predictions, but during training it always sees ground-truth tokens (Correct answer)
- The embedding layer is exposed to raw bytes instead of subword tokens
- Attention weights become biased toward early tokens in long sequences
Correct answer: At test time the model sees its own predictions, but during training it always sees ground-truth tokens
Exposure bias arises because teacher-forcing at train time hides prediction errors, making the model fragile to its own mistakes at inference.
Question 6: Which approach to low-resource NLP adds a small number of trainable parameters to a frozen pre-trained model?
- Full fine-tuning
- Adapter layers (Correct answer)
- Data augmentation
- Ensemble methods
Correct answer: Adapter layers
Adapter layers insert lightweight bottleneck modules between transformer layers and only those new parameters are trained, preserving the original weights.
Question 7: In coreference resolution, what is an 'antecedent'?
- The pronoun that refers to an entity
- The earlier mention that a pronoun or noun phrase refers back to (Correct answer)
- The head noun of a noun phrase
- A coreferent pair that spans sentence boundaries
Correct answer: The earlier mention that a pronoun or noun phrase refers back to
An antecedent is the previously mentioned entity to which a subsequent referring expression (like a pronoun) resolves.
NLP Advanced Topics & Theory 3
In seq2seq models, what problem does the attention mechanism primarily address?