NLP Tokenization 5 — Questions and Answers
Question 1: What is 'offset mapping' in tokenizer output and what is it used for?
- A lookup table mapping token IDs to their vector offsets in embedding matrices
- A list of (start, end) character positions in the original text corresponding to each token, used for span extraction tasks (Correct answer)
- The byte offset of each token in the serialized vocabulary file
- A mapping from subword tokens to their parent word indices only
Correct answer: A list of (start, end) character positions in the original text corresponding to each token, used for span extraction tasks
Offset mapping records the character-level start and end positions of each token in the original input string, enabling tasks like question answering to return answer spans.
Question 2: In the Hugging Face Tokenizers library, what does 'fast' vs. 'slow' tokenizer refer to?
- Fast tokenizers use GPU acceleration; slow tokenizers run on CPU only
- Fast tokenizers are implemented in Rust and support features like offset mapping; slow tokenizers are pure Python implementations (Correct answer)
- Fast tokenizers use BPE; slow tokenizers use word-level splitting
- Fast tokenizers skip normalization steps; slow tokenizers apply all preprocessing
Correct answer: Fast tokenizers are implemented in Rust and support features like offset mapping; slow tokenizers are pure Python implementations
Hugging Face 'fast' tokenizers are backed by the Rust-based tokenizers library, offering speed improvements and additional features like offset mapping not available in Python-based 'slow' tokenizers.
Question 3: What is 'multilingual tokenization' and what challenge does it address?
- Tokenizing only English text with multilingual labels
- Building a shared vocabulary that covers multiple languages so a single model can process text in many languages (Correct answer)
- Translating tokens from one language to another before processing
- Using a separate tokenizer for each language in the training data
Correct answer: Building a shared vocabulary that covers multiple languages so a single model can process text in many languages
Multilingual tokenization trains a single subword vocabulary on text from many languages, enabling cross-lingual transfer in models like mBERT and XLM-RoBERTa.
Question 4: What does 'tokenizer overfitting' mean in practice?
- The tokenizer memorizes all training sentences instead of learning subword rules
- A vocabulary trained on a domain-specific corpus that performs poorly on general text because its subwords are too specialized (Correct answer)
- The tokenizer produces too many tokens per sentence on the training set
- Applying the tokenizer more than once to the same text
Correct answer: A vocabulary trained on a domain-specific corpus that performs poorly on general text because its subwords are too specialized
A tokenizer trained on narrow domain text may learn highly domain-specific subword units that fragment out-of-domain text inefficiently.
Question 5: Which statement correctly describes how Chinese text is typically tokenized in BERT-based models?
- Chinese is tokenized exactly like English using whitespace splitting
- Each Chinese character is treated as an individual token since Chinese doesn't use whitespace between words (Correct answer)
- Chinese text is first converted to Pinyin before tokenization
- Chinese uses sentence-level tokenization only
Correct answer: Each Chinese character is treated as an individual token since Chinese doesn't use whitespace between words
Chinese BERT tokenization inserts spaces around every character before applying WordPiece, effectively treating each character as a basic unit.
Question 6: What is 'token budget' or 'token limit' and why does it matter for applications using large language models?
- The maximum number of distinct words allowed in user input
- The maximum number of tokens (input + output) a model can process in one request, affecting cost and what fits in context (Correct answer)
- The minimum number of tokens required for a model to generate coherent output
- The number of tokens the model reserves internally for its reasoning process
Correct answer: The maximum number of tokens (input + output) a model can process in one request, affecting cost and what fits in context
LLM APIs charge per token and enforce context window limits, so understanding token counts is critical for managing cost and ensuring inputs fit within the model's context.
Question 7: What is a 'tokenizer mismatch' and why is it a critical issue in NLP deployments?
- Using two different tokenization libraries that produce the same output
- Using a different tokenizer at inference time than was used during model training, causing the model to receive unexpected token ID sequences (Correct answer)
- Running the tokenizer on a different operating system than it was built on
- Applying tokenization to audio or image inputs instead of text
Correct answer: Using a different tokenizer at inference time than was used during model training, causing the model to receive unexpected token ID sequences
A tokenizer mismatch means the model receives token IDs that don't correspond to the embeddings it learned, causing degraded or nonsensical outputs.
NLP Tokenization 5
What is 'offset mapping' in tokenizer output and what is it used for?