Natural Language Processing Trivia 4 — Questions and Answers
Question 1: What is 'perplexity' used to measure in NLP language models?
- The diversity of vocabulary in a model's outputs
- How well a probability model predicts a sample; lower is better (Correct answer)
- The number of parameters in a neural language model
- The speed of text generation in tokens per second
Correct answer: How well a probability model predicts a sample; lower is better
Perplexity measures how well a language model predicts a held-out test set; a lower perplexity indicates the model assigns higher probability to the actual text.
Question 2: Which NLP concept refers to words appearing in similar contexts having similar meanings?
- Syntax hypothesis
- Distributional hypothesis (Correct answer)
- Pragmatics principle
- Morphological invariance
Correct answer: Distributional hypothesis
The distributional hypothesis states that words occurring in similar contexts tend to have similar meanings, forming the theoretical basis for word embeddings.
Question 3: In NLP, what does 'zero-shot learning' refer to?
- Training a model with no labeled data at all
- A model performing a task it was never explicitly trained on (Correct answer)
- Generating text with zero randomness (temperature=0)
- Removing all training examples before fine-tuning
Correct answer: A model performing a task it was never explicitly trained on
Zero-shot learning refers to a model's ability to perform tasks it was not explicitly trained on, relying on general language understanding and task descriptions.
Question 4: What is the role of the 'decoder' in a sequence-to-sequence (seq2seq) model?
- To encode the input sentence into a fixed vector
- To generate the output sequence token by token from the encoded representation (Correct answer)
- To compute attention weights over the input
- To tokenize the input text before processing
Correct answer: To generate the output sequence token by token from the encoded representation
In a seq2seq model, the decoder takes the encoder's representation and generates the output sequence (e.g., translated text) one token at a time.
Question 5: Which phenomenon occurs when an NLP model performs well on training data but poorly on new, unseen data?
- Underfitting
- Overfitting (Correct answer)
- Catastrophic forgetting
- Gradient vanishing
Correct answer: Overfitting
Overfitting occurs when a model learns the training data too specifically, including its noise, causing degraded performance on new, unseen examples.
Question 6: What is 'coreference resolution' in NLP?
- Translating pronouns between languages
- Identifying all expressions in text that refer to the same real-world entity (Correct answer)
- Resolving ambiguity in named entities
- Matching question phrases to answer spans
Correct answer: Identifying all expressions in text that refer to the same real-world entity
Coreference resolution determines which words or phrases in a text refer to the same entity, such as linking 'she' back to 'Mary' mentioned earlier.
Question 7: Which pre-training objective does BERT use to learn bidirectional representations?
- Next sentence prediction only
- Masked language modeling and next sentence prediction (Correct answer)
- Causal language modeling left-to-right
- Contrastive sentence embedding
Correct answer: Masked language modeling and next sentence prediction
BERT pre-trains using masked language modeling (predicting randomly masked tokens) and next sentence prediction (determining if two sentences are consecutive).
Natural Language Processing Trivia 4
What is 'perplexity' used to measure in NLP language models?