NLP Advanced Topics & Theory 2 — Questions and Answers
Question 1: What is the primary purpose of the Transformer's multi-head attention mechanism?
- To reduce model size by pruning redundant heads
- To allow the model to jointly attend to information from different representation subspaces (Correct answer)
- To speed up sequential token generation
- To replace positional encodings with learned embeddings
Correct answer: To allow the model to jointly attend to information from different representation subspaces
Multi-head attention runs attention in parallel across multiple subspaces, letting the model capture different types of relationships simultaneously.
Question 2: Which decoding strategy samples from the top-k most probable next tokens at each step?
- Beam search
- Greedy decoding
- Top-k sampling (Correct answer)
- Temperature scaling
Correct answer: Top-k sampling
Top-k sampling restricts the sampling pool to the k highest-probability tokens, balancing diversity and coherence.
Question 3: In the context of NLP, what does 'perplexity' measure?
- The number of unique tokens in a corpus
- How well a probability model predicts a sample (Correct answer)
- The depth of a parse tree
- The cosine similarity between two sentence embeddings
Correct answer: How well a probability model predicts a sample
Perplexity is the exponentiated average negative log-likelihood of a test set, measuring how uncertain the model is about each token.
Question 4: What is 'catastrophic forgetting' in continual learning for NLP models?
- A model forgetting stop-word rules during fine-tuning
- The tendency of a neural network to lose previously learned information when trained on new tasks (Correct answer)
- Gradient explosion during multi-task training
- A tokenizer dropping rare tokens from its vocabulary
Correct answer: The tendency of a neural network to lose previously learned information when trained on new tasks
Catastrophic forgetting occurs when optimizing for a new task overwrites the weights that encoded prior knowledge.
Question 5: Which technique uses a smaller 'student' model to mimic the output distribution of a larger 'teacher' model?
- Pruning
- Quantization
- Knowledge distillation (Correct answer)
- Low-rank factorization
Correct answer: Knowledge distillation
Knowledge distillation trains a compact student model to match the soft probability outputs of a pre-trained teacher, compressing knowledge without large accuracy loss.
Question 6: What is the role of the key-query-value (K-Q-V) structure in self-attention?
- Keys encode positional info, queries encode tokens, values encode grammar
- Queries match against keys to produce attention weights, then weighted sum of values forms the output (Correct answer)
- Values store embeddings, keys store loss gradients, queries store hidden states
- All three are identical projections used for numerical stability
Correct answer: Queries match against keys to produce attention weights, then weighted sum of values forms the output
Each token's query attends over all keys; the resulting attention weights are applied to values to produce a context-aware representation.
Question 7: Which regularization technique randomly masks input tokens during pre-training, requiring the model to reconstruct them?
- Dropout
- Weight decay
- Masked language modeling (MLM) (Correct answer)
- Label smoothing
Correct answer: Masked language modeling (MLM)
MLM, used in BERT, masks 15% of tokens and trains the model to predict the originals, learning bidirectional context.
NLP Advanced Topics & Theory 2
What is the primary purpose of the Transformer's multi-head attention mechanism?