NLP Tokenization 5

Question 1

What is 'offset mapping' in tokenizer output and what is it used for?

Accepted Answer

A list of (start, end) character positions in the original text corresponding to each token, used for span extraction tasks

Answer

Offset mapping records the character-level start and end positions of each token in the original input string, enabling tasks like question answering to return answer spans.

Question 2

In the Hugging Face Tokenizers library, what does 'fast' vs. 'slow' tokenizer refer to?

Accepted Answer

Fast tokenizers are implemented in Rust and support features like offset mapping; slow tokenizers are pure Python implementations

Answer

Hugging Face 'fast' tokenizers are backed by the Rust-based tokenizers library, offering speed improvements and additional features like offset mapping not available in Python-based 'slow' tokenizers.

Question 3

What is 'multilingual tokenization' and what challenge does it address?

Accepted Answer

Building a shared vocabulary that covers multiple languages so a single model can process text in many languages

Answer

Multilingual tokenization trains a single subword vocabulary on text from many languages, enabling cross-lingual transfer in models like mBERT and XLM-RoBERTa.

Question 4

What does 'tokenizer overfitting' mean in practice?

Accepted Answer

A vocabulary trained on a domain-specific corpus that performs poorly on general text because its subwords are too specialized

Answer

A tokenizer trained on narrow domain text may learn highly domain-specific subword units that fragment out-of-domain text inefficiently.

Question 5

Which statement correctly describes how Chinese text is typically tokenized in BERT-based models?

Accepted Answer

Each Chinese character is treated as an individual token since Chinese doesn't use whitespace between words

Answer

Chinese BERT tokenization inserts spaces around every character before applying WordPiece, effectively treating each character as a basic unit.

NLP Practice Test

NLP Practice Test

NLP Tokenization 5