Natural Language Processing Trivia 5 — Questions and Answers
Question 1: What is 'transfer learning' in the context of NLP?
- Moving a trained model from one server to another
- Applying knowledge gained from pre-training on large data to specific downstream tasks (Correct answer)
- Transferring labels from one dataset to another
- Sharing model weights between two different architectures
Correct answer: Applying knowledge gained from pre-training on large data to specific downstream tasks
Transfer learning in NLP involves pre-training a model on large unlabeled text, then fine-tuning it on smaller labeled datasets for specific tasks.
Question 2: What does 'hallucination' mean when describing large language model outputs?
- Generating text with very high confidence scores
- Producing plausible-sounding but factually incorrect or fabricated information (Correct answer)
- Repeating the same phrase multiple times
- Refusing to answer a question due to safety filters
Correct answer: Producing plausible-sounding but factually incorrect or fabricated information
Hallucination in LLMs refers to the model generating confident-sounding but factually incorrect, fabricated, or nonsensical information.
Question 3: In NLP, what is the 'vocabulary' of a language model?
- The set of all documents used for training
- The fixed set of tokens the model can process and generate (Correct answer)
- The list of hyperparameters used during training
- The collection of labeled examples in the training set
Correct answer: The fixed set of tokens the model can process and generate
The vocabulary is the fixed set of tokens (words, subwords, or characters) that a language model recognizes and can produce as output.
Question 4: Which subword tokenization algorithm is used by GPT-2 and later GPT models?
- WordPiece
- Byte-Pair Encoding (BPE) (Correct answer)
- Unigram Language Model
- SentencePiece character model
Correct answer: Byte-Pair Encoding (BPE)
GPT models use Byte-Pair Encoding (BPE), which iteratively merges the most frequent byte pairs to build a subword vocabulary.
Question 5: What is the main advantage of subword tokenization over word-level tokenization?
- It produces shorter sequences, speeding up training
- It handles rare and out-of-vocabulary words by splitting them into known subword units (Correct answer)
- It eliminates the need for a fixed vocabulary
- It preserves full sentence structure better than character models
Correct answer: It handles rare and out-of-vocabulary words by splitting them into known subword units
Subword tokenization breaks unknown or rare words into smaller subword pieces that are in the vocabulary, avoiding out-of-vocabulary problems entirely.
Question 6: What NLP task involves automatically summarizing the key points of a longer document?
- Text classification
- Automatic text summarization (Correct answer)
- Relation extraction
- Semantic textual similarity
Correct answer: Automatic text summarization
Automatic text summarization is the NLP task of producing a concise, coherent summary of a longer document, either by extracting key sentences or generating new text.
Question 7: In transformer models, what does 'positional encoding' solve?
- It adds word frequency information to embeddings
- It gives the model information about the order of tokens, since attention itself is order-agnostic (Correct answer)
- It encodes part-of-speech tags into input vectors
- It normalizes token embeddings to unit length
Correct answer: It gives the model information about the order of tokens, since attention itself is order-agnostic
Since the self-attention mechanism treats all positions equally, positional encodings are added to token embeddings so the model can distinguish word order.
Natural Language Processing Trivia 5
What is 'transfer learning' in the context of NLP?