NLP Language Models 3

Question 1

What is the 'context window' of a transformer-based language model?

Accepted Answer

The maximum number of input tokens the model can attend to during inference

Answer

The context window defines the maximum sequence length the model can process as input, limiting how much prior text it can condition on.

Question 2

In the Transformer architecture, what is the role of positional encodings?

Accepted Answer

They inject information about the position of each token since self-attention is order-agnostic

Answer

Since self-attention treats input tokens as a set (no inherent order), positional encodings add token position information so the model can learn order-dependent patterns.

Question 3

What is 'nucleus sampling' (top-p sampling) in language model decoding?

Accepted Answer

Sampling from the smallest set of tokens whose cumulative probability exceeds a threshold p

Answer

Top-p sampling dynamically selects a minimal set of tokens whose probabilities sum to at least p, then samples from that set — balancing diversity and coherence.

Question 4

What does 'instruction fine-tuning' accomplish in large language models?

Accepted Answer

It trains the model on diverse instruction-response pairs to improve its ability to follow user directions

Answer

Instruction fine-tuning teaches a pretrained model to interpret and respond to natural language instructions across a wide range of tasks.

Question 5

Which of the following is a major limitation of n-gram language models compared to neural language models?

Accepted Answer

N-gram models suffer from data sparsity and cannot generalize across similar words

Answer

N-gram models treat words as discrete symbols and cannot generalize — 'dog ran' and 'puppy ran' are completely unrelated — while neural models use dense embeddings that capture similarity.

NLP Practice Test

NLP Practice Test

NLP Language Models 3