NLP Tokenization 3 — Questions and Answers
Question 1: What problem does the SentencePiece library solve that earlier tokenizers did not?
- It eliminates the need for any vocabulary
- It performs tokenization without relying on language-specific whitespace rules, treating raw text as input (Correct answer)
- It automatically detects the language of the input text
- It converts tokens to dense vector embeddings
Correct answer: It performs tokenization without relying on language-specific whitespace rules, treating raw text as input
SentencePiece treats the input as a raw stream of Unicode characters, making it language-agnostic and not dependent on whitespace-delimited words.
Question 2: In tokenization, what is a 'token ID' (or input ID)?
- A floating-point embedding vector for a token
- The integer index of a token in the tokenizer's vocabulary (Correct answer)
- The Unicode code point of the first character in a token
- The position of a token in the sentence
Correct answer: The integer index of a token in the tokenizer's vocabulary
A token ID is the integer index that maps a token to its row in the model's embedding matrix.
Question 3: What is 'detokenization' in NLP?
- Removing stopwords from a token sequence
- Converting a sequence of tokens back into a human-readable string (Correct answer)
- Splitting tokens into individual characters
- Assigning part-of-speech tags to tokens
Correct answer: Converting a sequence of tokens back into a human-readable string
Detokenization reconstructs the original (or near-original) text string from a list of tokens, reversing the tokenization process.
Question 4: Which tokenization approach is used by GPT-2 and GPT-3?
- WordPiece
- Unigram language model
- Byte-level BPE (Correct answer)
- Character n-gram tokenization
Correct answer: Byte-level BPE
GPT-2 and GPT-3 use byte-level BPE, which operates on UTF-8 bytes rather than Unicode characters, ensuring every string can be tokenized.
Question 5: What does 'attention mask' indicate in the output of a Hugging Face tokenizer?
- Which tokens should be masked for masked language modeling
- Which token positions the model should attend to (1) vs. ignore as padding (0) (Correct answer)
- The self-attention weights computed by the transformer
- The probability of each token given the context
Correct answer: Which token positions the model should attend to (1) vs. ignore as padding (0)
The attention mask is a binary tensor marking real tokens with 1 and padding tokens with 0, telling the model which positions to attend to.
Question 6: What is the 'unknown token' ([UNK]) used for in word-level tokenization?
- To represent the beginning of a sentence
- To replace any word not found in the vocabulary during inference (Correct answer)
- To separate two sentences in a paired input
- To indicate a masked position during pre-training
Correct answer: To replace any word not found in the vocabulary during inference
The [UNK] token is a fallback that replaces any input word absent from the fixed vocabulary, grouping all OOV words into a single representation.
Question 7: When using subword tokenization, the word 'unbelievably' might be split into which of the following?
- ['u', 'n', 'b', 'e', 'l', 'i', 'e', 'v', 'a', 'b', 'l', 'y']
- ['un', '##believ', '##ably'] (Correct answer)
- ['unbelievably'] as a single token always
- ['un', 'believe', '-', 'ably']
Correct answer: ['un', '##believ', '##ably']
WordPiece-style subword tokenizers break words into frequent subword pieces, using '##' to indicate continuation of the previous token.
NLP Tokenization 3
What problem does the SentencePiece library solve that earlier tokenizers did not?