NLP Tokenization 4 — Questions and Answers
Question 1: What is the 'unigram language model' tokenization algorithm used in SentencePiece?
- An algorithm that always picks the longest matching token from the vocabulary
- A probabilistic approach that finds the tokenization maximizing the likelihood under a unigram language model, iteratively pruning the vocabulary (Correct answer)
- An algorithm that splits text on every space and punctuation character
- A method that assigns equal probability to all possible tokenizations
Correct answer: A probabilistic approach that finds the tokenization maximizing the likelihood under a unigram language model, iteratively pruning the vocabulary
The unigram algorithm starts with a large vocabulary and iteratively removes tokens that least reduce the corpus likelihood until the target vocabulary size is reached.
Question 2: What is 'truncation' in the context of tokenizer preprocessing for transformer models?
- Removing low-frequency tokens from the vocabulary
- Cutting token sequences that exceed the model's maximum input length (Correct answer)
- Replacing tokens with their stemmed form
- Splitting a document into overlapping windows of tokens
Correct answer: Cutting token sequences that exceed the model's maximum input length
Truncation clips token sequences to fit within the model's maximum context window, discarding tokens beyond the limit.
Question 3: What is 'padding' in batch tokenization and why is it necessary?
- Adding noise tokens to prevent overfitting
- Adding [PAD] tokens to shorter sequences so all sequences in a batch have the same length, enabling efficient tensor operations (Correct answer)
- Repeating the last token to fill unused positions
- Inserting separator tokens between sentences
Correct answer: Adding [PAD] tokens to shorter sequences so all sequences in a batch have the same length, enabling efficient tensor operations
Padding equalizes sequence lengths within a batch so they can be stacked into rectangular tensors required by GPU-accelerated matrix operations.
Question 4: Which of the following is a key advantage of character-level tokenization over word-level tokenization?
- Shorter sequence lengths for the same text
- No out-of-vocabulary (OOV) problem since any text can be represented (Correct answer)
- Better capture of semantic meaning per token
- Smaller model input sizes
Correct answer: No out-of-vocabulary (OOV) problem since any text can be represented
Character-level tokenization can represent any string using a small fixed alphabet, completely eliminating OOV issues.
Question 5: What does 'token alignment' refer to when tokenizing text for tasks like Named Entity Recognition (NER)?
- Sorting tokens by their frequency in the corpus
- Mapping subword tokens back to their original word boundaries to correctly assign labels (Correct answer)
- Aligning the vocabulary of two different tokenizers
- Ensuring all tokens have the same vector length
Correct answer: Mapping subword tokens back to their original word boundaries to correctly assign labels
In NER, word-level labels must be aligned to subword tokens; typically only the first subword of each word receives the label while others get a special ignore label.
Question 6: What is the effect of choosing a larger vocabulary size in subword tokenization?
- Shorter token sequences but more unique token types to learn embeddings for (Correct answer)
- Longer token sequences and fewer unique token types
- No effect on sequence length or vocabulary coverage
- Shorter token sequences and fewer unique token types
Correct answer: Shorter token sequences but more unique token types to learn embeddings for
A larger vocabulary allows more complete words and longer subwords, reducing sequence length but requiring the model to learn more embedding vectors.
Question 7: Why might a tokenizer produce different numbers of tokens for the same English word depending on whether it appears at the start of a sentence or mid-sentence in GPT-style BPE?
- Sentence position does not affect tokenization in any tokenizer
- GPT-style BPE treats a leading space as part of the token, so 'dog' and ' dog' are different tokens (Correct answer)
- The tokenizer uses context from surrounding words to re-segment tokens
- Capital letters at the start cause an extra token to be emitted
Correct answer: GPT-style BPE treats a leading space as part of the token, so 'dog' and ' dog' are different tokens
GPT tokenizers prepend a space to most words as a prefix (e.g., 'Ġdog'), distinguishing a word at the start of a sentence from a mid-sentence occurrence.
NLP Tokenization 4
What is the 'unigram language model' tokenization algorithm used in SentencePiece?