NLP Language Models 3 — Questions and Answers
Question 1: What is the 'context window' of a transformer-based language model?
- The number of output tokens the model can generate at once
- The maximum number of input tokens the model can attend to during inference (Correct answer)
- The sliding window used to batch training examples
- The number of attention heads in the final layer
Correct answer: The maximum number of input tokens the model can attend to during inference
The context window defines the maximum sequence length the model can process as input, limiting how much prior text it can condition on.
Question 2: In the Transformer architecture, what is the role of positional encodings?
- They replace token embeddings to reduce memory usage
- They inject information about the position of each token since self-attention is order-agnostic (Correct answer)
- They mask future tokens during decoder training
- They normalize attention weights across all heads
Correct answer: They inject information about the position of each token since self-attention is order-agnostic
Since self-attention treats input tokens as a set (no inherent order), positional encodings add token position information so the model can learn order-dependent patterns.
Question 3: What is 'nucleus sampling' (top-p sampling) in language model decoding?
- Sampling only from the single most probable next token at each step
- Sampling from the smallest set of tokens whose cumulative probability exceeds a threshold p (Correct answer)
- Sampling uniformly from the top-p% of the vocabulary by frequency
- Applying beam search with p beams simultaneously
Correct answer: Sampling from the smallest set of tokens whose cumulative probability exceeds a threshold p
Top-p sampling dynamically selects a minimal set of tokens whose probabilities sum to at least p, then samples from that set — balancing diversity and coherence.
Question 4: What does 'instruction fine-tuning' accomplish in large language models?
- It reduces the model's parameter count by pruning unused weights
- It trains the model on diverse instruction-response pairs to improve its ability to follow user directions (Correct answer)
- It replaces pretraining with supervised learning on a single task
- It compresses the model's embeddings to fit on edge devices
Correct answer: It trains the model on diverse instruction-response pairs to improve its ability to follow user directions
Instruction fine-tuning teaches a pretrained model to interpret and respond to natural language instructions across a wide range of tasks.
Question 5: Which of the following is a major limitation of n-gram language models compared to neural language models?
- N-gram models cannot assign probabilities to unseen sentences
- N-gram models suffer from data sparsity and cannot generalize across similar words (Correct answer)
- N-gram models require GPU hardware that was unavailable before 2010
- N-gram models only work with character-level tokenization
Correct answer: N-gram models suffer from data sparsity and cannot generalize across similar words
N-gram models treat words as discrete symbols and cannot generalize — 'dog ran' and 'puppy ran' are completely unrelated — while neural models use dense embeddings that capture similarity.
Question 6: What is 'beam search' in language model generation, and what is its main drawback?
- Beam search samples randomly from top-k tokens; its drawback is high variance
- Beam search maintains multiple candidate sequences simultaneously; it tends to produce generic, repetitive outputs (Correct answer)
- Beam search restricts generation to a fixed vocabulary beam; it cannot handle rare words
- Beam search uses attention beams to process long documents; it is slow on short sequences
Correct answer: Beam search maintains multiple candidate sequences simultaneously; it tends to produce generic, repetitive outputs
Beam search tracks the top-b sequences at each step by score, but the highest-scoring sequences often converge to safe, repetitive outputs rather than diverse ones.
Question 7: What is the primary function of the feed-forward sublayer within each Transformer block?
- To compute attention scores between all token pairs in the sequence
- To apply a position-wise nonlinear transformation independently to each token representation (Correct answer)
- To merge outputs from multiple attention heads into a single vector
- To encode positional information into the token embeddings
Correct answer: To apply a position-wise nonlinear transformation independently to each token representation
The feed-forward sublayer applies two linear transformations with a nonlinearity (e.g., ReLU/GELU) independently to each token, adding representational capacity beyond what attention provides.
NLP Language Models 3
What is the 'context window' of a transformer-based language model?