NLP Language Models 2

Question 1

What does 'perplexity' measure in the context of language models?

Accepted Answer

How well a probability model predicts a sample

Answer

Perplexity measures how well a language model predicts a held-out test set — lower perplexity indicates better predictive performance.

Question 2

Which smoothing technique redistributes probability mass from seen to unseen n-grams using interpolation across multiple n-gram orders?

Accepted Answer

Jelinek-Mercer smoothing

Answer

Jelinek-Mercer smoothing linearly interpolates probabilities from different n-gram orders to handle unseen n-grams.

Question 3

In a neural language model, what is the purpose of weight tying between the input embedding and output projection matrices?

Accepted Answer

To reduce parameter count while often improving performance

Answer

Weight tying shares the same matrix for input embeddings and the final output projection, reducing parameters and often improving generalization.

Question 4

What is the key distinction between an autoregressive language model and a masked language model?

Accepted Answer

Autoregressive models predict tokens left-to-right; masked models predict randomly masked tokens using full context

Answer

Autoregressive models like GPT generate tokens sequentially left-to-right, while masked models like BERT predict masked tokens using both left and right context.

Question 5

What problem does the 'vanishing gradient' cause specifically in recurrent language models?

Accepted Answer

The model fails to learn long-range dependencies because gradients shrink exponentially over time steps

Answer

Vanishing gradients cause the error signal to shrink as it propagates back through many time steps, preventing the model from learning dependencies between distant words.

NLP Practice Test

NLP Practice Test

NLP Language Models 2