NLP Language Models 5 — Questions and Answers
Question 1: What is 'quantization' in the deployment of large language models?
- Reducing the sequence length of inputs to lower memory footprint
- Representing model weights with lower-precision numbers (e.g., INT8 instead of FP32) to reduce size and speed up inference (Correct answer)
- Splitting the model across multiple GPUs to enable parallel decoding
- Pruning the model by removing attention heads with low average activation
Correct answer: Representing model weights with lower-precision numbers (e.g., INT8 instead of FP32) to reduce size and speed up inference
Quantization compresses model weights by using fewer bits per parameter, significantly reducing memory and improving inference speed with minimal accuracy loss.
Question 2: What is 'knowledge distillation' in the context of language models?
- Extracting factual knowledge from a model by querying it with structured prompts
- Training a smaller 'student' model to mimic the output distribution of a larger 'teacher' model (Correct answer)
- Combining knowledge from multiple pretrained models via ensemble averaging
- Distilling a model's training data into a smaller curated dataset for efficiency
Correct answer: Training a smaller 'student' model to mimic the output distribution of a larger 'teacher' model
Knowledge distillation trains a compact student model using the soft probability outputs (logits) of a larger teacher model as training targets, transferring its learned representations.
Question 3: What is the 'hallucination' problem in large language models?
- The model generates text that is fluent and confident but factually incorrect or fabricated (Correct answer)
- The model repeats phrases from the training data verbatim, violating copyright
- The model fails to generate coherent text when given ambiguous prompts
- The model's attention scores become undefined (NaN) on very long inputs
Correct answer: The model generates text that is fluent and confident but factually incorrect or fabricated
Hallucination refers to language models generating plausible-sounding but false information, a key reliability challenge since models optimize for fluency, not factual accuracy.
Question 4: In language modeling, what is the difference between 'top-k sampling' and 'greedy decoding'?
- Greedy decoding always selects the most probable token; top-k samples from the k most probable tokens (Correct answer)
- Top-k selects the k least probable tokens to maximize diversity; greedy uses the full vocabulary
- Greedy decoding uses beam search internally; top-k does not use any search
- Top-k sampling is deterministic; greedy decoding is stochastic
Correct answer: Greedy decoding always selects the most probable token; top-k samples from the k most probable tokens
Greedy decoding always picks the single highest-probability token (deterministic), while top-k restricts sampling to the k most probable tokens, introducing controlled randomness.
Question 5: What is a 'mixture of experts' (MoE) architecture in the context of large language models?
- An ensemble of independently trained models that vote on the final output
- A model architecture where each input token is routed to only a subset of specialized sub-networks (experts) rather than all parameters (Correct answer)
- A training strategy where multiple human experts annotate different portions of the dataset
- A multi-agent system where different models handle different languages
Correct answer: A model architecture where each input token is routed to only a subset of specialized sub-networks (experts) rather than all parameters
In MoE models, a learned gating network routes each token to a sparse subset of 'expert' feed-forward layers, enabling very large total parameter counts without proportionally increasing compute.
Question 6: What does 'grounding' a language model mean in NLP system design?
- Connecting the model's outputs to verifiable external knowledge or real-world data to reduce hallucination (Correct answer)
- Training the model on grounded (labeled) data instead of raw web text
- Fixing the model's weights so they cannot be updated during inference
- Limiting the model's vocabulary to domain-specific terms to improve precision
Correct answer: Connecting the model's outputs to verifiable external knowledge or real-world data to reduce hallucination
Grounding connects model outputs to external knowledge sources (databases, documents, APIs) so responses are anchored in verifiable facts rather than solely the model's parametric memory.
Question 7: What is 'tokenization' in language models, and why is byte-pair encoding (BPE) commonly used?
- Tokenization splits text into sentences; BPE is used because sentences are the natural unit of meaning
- Tokenization splits text into subword units; BPE iteratively merges frequent character pairs to balance vocabulary size with coverage of rare words (Correct answer)
- Tokenization converts text to integers; BPE assigns each character a unique prime number
- Tokenization removes stop words; BPE reinserts them after processing for grammatical correctness
Correct answer: Tokenization splits text into subword units; BPE iteratively merges frequent character pairs to balance vocabulary size with coverage of rare words
BPE learns a vocabulary of subword units by iteratively merging the most frequent adjacent character pairs, enabling models to handle rare and out-of-vocabulary words without an explosion in vocabulary size.
NLP Language Models 5
What is 'quantization' in the deployment of large language models?