MS-DS Master of Data science Natural Language Processing 5 — Questions and Answers
Question 1: In coreference resolution, what does it mean for two mentions to be 'coreferent'?
- They share the same syntactic head word
- They refer to the same entity or event in the world (Correct answer)
- They appear within the same sentence boundary
- They have identical surface string forms
Correct answer: They refer to the same entity or event in the world
Coreference resolution links noun phrases (e.g., 'Barack Obama' and 'he') that refer to the same real-world entity, forming coreference chains across a document.
Question 2: What distinguishes nucleus sampling (top-p sampling) from top-k sampling in text generation?
- Nucleus sampling uses beam search internally while top-k does not
- Nucleus sampling dynamically selects the smallest set of tokens whose cumulative probability exceeds threshold p, adapting to the distribution shape (Correct answer)
- Top-k sampling is deterministic; nucleus sampling is stochastic
- Nucleus sampling requires a separate reward model to rank tokens
Correct answer: Nucleus sampling dynamically selects the smallest set of tokens whose cumulative probability exceeds threshold p, adapting to the distribution shape
Unlike top-k which fixes the number of candidates, top-p sampling includes tokens until their cumulative probability reaches p, allowing a larger or smaller candidate set depending on distribution sharpness.
Question 3: In multi-task learning for NLP, what is the key assumption that justifies sharing parameters across tasks?
- All tasks use the same label set
- Tasks share useful linguistic features in their input representations, enabling positive transfer (Correct answer)
- Tasks must have the same training set size
- Shared parameters always reduce total model size below single-task baselines
Correct answer: Tasks share useful linguistic features in their input representations, enabling positive transfer
Multi-task learning assumes that related NLP tasks (e.g., POS tagging and NER) rely on overlapping syntactic or semantic features, so shared lower layers transfer knowledge beneficially.
Question 4: Which of the following is a key limitation of the BLEU metric when evaluating natural language generation quality?
- BLEU requires human annotators for every evaluation run
- BLEU ignores recall and does not correlate well with human judgments for fluency or meaning preservation (Correct answer)
- BLEU can only evaluate translations between English and French
- BLEU requires the generated output to be the same length as the reference
Correct answer: BLEU ignores recall and does not correlate well with human judgments for fluency or meaning preservation
BLEU is a precision-focused n-gram overlap metric that ignores synonymy, paraphrase, and sentence-level meaning, resulting in poor correlation with human quality judgments on many generation tasks.
Question 5: What is 'catastrophic forgetting' in the context of sequential fine-tuning of pre-trained language models?
- The model forgets the fine-tuning task after each training epoch restart
- The model's performance on previously learned tasks degrades significantly when fine-tuned on a new task (Correct answer)
- The model loses its ability to handle long sequences after fine-tuning on short ones
- Gradient checkpointing causes the model to discard intermediate activations
Correct answer: The model's performance on previously learned tasks degrades significantly when fine-tuned on a new task
When a neural network is fine-tuned on a new task, the parameter updates can overwrite knowledge from prior tasks, a phenomenon called catastrophic forgetting or catastrophic interference.
Question 6: In the encoder-decoder architecture for neural machine translation, what is the function of the cross-attention mechanism in the decoder?
- It attends to previously generated target tokens to enforce fluency
- It allows the decoder to selectively attend to relevant encoder hidden states when generating each target token (Correct answer)
- It computes attention between source tokens to build richer source representations
- It masks future encoder positions to prevent data leakage
Correct answer: It allows the decoder to selectively attend to relevant encoder hidden states when generating each target token
Cross-attention in the decoder uses decoder query vectors against encoder key-value pairs, enabling the model to focus on the most relevant source tokens at each generation step.
Question 7: Which regularization technique specific to transformer training helps stabilize optimization by normalizing activations before each sublayer's computation?
- Dropout applied to attention weights only
- Pre-layer normalization (Pre-LN) applied before each sublayer (Correct answer)
- L2 weight decay on query and key matrices
- Gradient clipping with a norm threshold of 1.0
Correct answer: Pre-layer normalization (Pre-LN) applied before each sublayer
Pre-LN applies layer normalization to the input of each sublayer (attention or FFN) before the residual addition, leading to more stable gradient flow compared to the original Post-LN Transformer.
MS-DS Master of Data science Natural Language Processing 5
In coreference resolution, what does it mean for two mentions to be 'coreferent'?