NLP Language Models 2 — Questions and Answers
Question 1: What does 'perplexity' measure in the context of language models?
- The number of parameters in the model
- How well a probability model predicts a sample (Correct answer)
- The vocabulary size used during training
- The length of the context window
Correct answer: How well a probability model predicts a sample
Perplexity measures how well a language model predicts a held-out test set — lower perplexity indicates better predictive performance.
Question 2: Which smoothing technique redistributes probability mass from seen to unseen n-grams using interpolation across multiple n-gram orders?
- Laplace smoothing
- Kneser-Ney smoothing
- Good-Turing smoothing
- Jelinek-Mercer smoothing (Correct answer)
Correct answer: Jelinek-Mercer smoothing
Jelinek-Mercer smoothing linearly interpolates probabilities from different n-gram orders to handle unseen n-grams.
Question 3: In a neural language model, what is the purpose of weight tying between the input embedding and output projection matrices?
- To reduce overfitting by increasing model complexity
- To reduce parameter count while often improving performance (Correct answer)
- To ensure the model outputs only valid vocabulary tokens
- To speed up backpropagation through time
Correct answer: To reduce parameter count while often improving performance
Weight tying shares the same matrix for input embeddings and the final output projection, reducing parameters and often improving generalization.
Question 4: What is the key distinction between an autoregressive language model and a masked language model?
- Autoregressive models use bidirectional context; masked models use unidirectional context
- Autoregressive models predict tokens left-to-right; masked models predict randomly masked tokens using full context (Correct answer)
- Autoregressive models require more training data than masked models
- Masked models cannot be used for text generation tasks
Correct answer: Autoregressive models predict tokens left-to-right; masked models predict randomly masked tokens using full context
Autoregressive models like GPT generate tokens sequentially left-to-right, while masked models like BERT predict masked tokens using both left and right context.
Question 5: What problem does the 'vanishing gradient' cause specifically in recurrent language models?
- The model cannot process sequences longer than its hidden state size
- The model fails to learn long-range dependencies because gradients shrink exponentially over time steps (Correct answer)
- The model overfits to frequent words and ignores rare vocabulary
- The model produces repetitive outputs because gradients become too large
Correct answer: The model fails to learn long-range dependencies because gradients shrink exponentially over time steps
Vanishing gradients cause the error signal to shrink as it propagates back through many time steps, preventing the model from learning dependencies between distant words.
Question 6: Which of the following best describes 'zero-shot' capability in large language models?
- The model was trained on zero examples of a task but can still perform it using instructions alone (Correct answer)
- The model requires zero fine-tuning steps after pretraining
- The model achieves zero perplexity on its training set
- The model uses zero attention heads for efficiency
Correct answer: The model was trained on zero examples of a task but can still perform it using instructions alone
Zero-shot capability means a model can perform a task it was never explicitly trained on, guided only by a natural language instruction or prompt.
Question 7: What is 'temperature' in language model text generation, and what does setting it to 0 do?
- Temperature controls training speed; setting it to 0 stops training
- Temperature scales logits before softmax; setting it to 0 makes generation deterministic by always picking the most probable token (Correct answer)
- Temperature controls vocabulary size; setting it to 0 uses all tokens equally
- Temperature adjusts context length; setting it to 0 disables the context window
Correct answer: Temperature scales logits before softmax; setting it to 0 makes generation deterministic by always picking the most probable token
Temperature divides logits before the softmax — lower values sharpen the distribution, and temperature=0 (greedy decoding) always selects the highest-probability token.
NLP Language Models 2
What does 'perplexity' measure in the context of language models?