NLP Tokenization 2 — Questions and Answers
Question 1: What is the primary purpose of Byte Pair Encoding (BPE) in tokenization?
- To split text only on whitespace boundaries
- To iteratively merge the most frequent adjacent byte pairs into a single token (Correct answer)
- To assign each character a unique prime number identifier
- To convert all text to lowercase before tokenization
Correct answer: To iteratively merge the most frequent adjacent byte pairs into a single token
BPE starts with individual characters and iteratively merges the most frequent adjacent pairs to build a vocabulary of subword units.
Question 2: Which tokenization strategy is best suited for handling out-of-vocabulary (OOV) words?
- Word-level tokenization
- Sentence-level tokenization
- Subword tokenization (Correct answer)
- Paragraph-level tokenization
Correct answer: Subword tokenization
Subword tokenization breaks unknown words into known subword pieces, effectively eliminating the OOV problem.
Question 3: In the context of tokenization, what does the term 'vocabulary size' refer to?
- The number of sentences in the training corpus
- The total number of unique tokens the tokenizer can produce (Correct answer)
- The average number of tokens per document
- The maximum length of a single token in characters
Correct answer: The total number of unique tokens the tokenizer can produce
Vocabulary size is the count of distinct tokens (words, subwords, or characters) that a tokenizer's model recognizes.
Question 4: Which tokenizer is used by the original BERT model?
- SentencePiece with unigram model
- Byte Pair Encoding (BPE)
- WordPiece (Correct answer)
- Character-level tokenization
Correct answer: WordPiece
BERT uses WordPiece tokenization, which builds a vocabulary by merging token pairs that maximize the likelihood of the training data.
Question 5: What distinguishes WordPiece from BPE in subword tokenization?
- WordPiece merges based on frequency, BPE merges based on likelihood
- BPE merges based on frequency, WordPiece merges based on likelihood of the training data (Correct answer)
- WordPiece only operates on characters, BPE operates on words
- BPE uses a fixed vocabulary while WordPiece uses a dynamic vocabulary
Correct answer: BPE merges based on frequency, WordPiece merges based on likelihood of the training data
BPE selects the most frequent pair to merge, while WordPiece selects the pair whose merge maximizes the training corpus likelihood.
Question 6: What is a 'special token' in the context of modern NLP tokenizers?
- A token that appears only once in the entire corpus
- A reserved token with a specific role such as [CLS], [SEP], or [PAD] (Correct answer)
- The longest token in the vocabulary
- A token representing punctuation marks
Correct answer: A reserved token with a specific role such as [CLS], [SEP], or [PAD]
Special tokens like [CLS], [SEP], and [PAD] are reserved markers added by tokenizers to encode structural information for models.
Question 7: Which of the following best describes 'tokenization normalization'?
- Sorting all tokens alphabetically before processing
- Preprocessing steps like lowercasing or accent removal applied before splitting text into tokens (Correct answer)
- Computing the frequency of each token in the corpus
- Converting tokens back to their original text form
Correct answer: Preprocessing steps like lowercasing or accent removal applied before splitting text into tokens
Normalization refers to preprocessing transformations—such as Unicode normalization, lowercasing, or accent stripping—applied to text before tokenization.
NLP Tokenization 2
What is the primary purpose of Byte Pair Encoding (BPE) in tokenization?