NLP Tokenization 4

Question 1

What is the 'unigram language model' tokenization algorithm used in SentencePiece?

Accepted Answer

A probabilistic approach that finds the tokenization maximizing the likelihood under a unigram language model, iteratively pruning the vocabulary

Answer

The unigram algorithm starts with a large vocabulary and iteratively removes tokens that least reduce the corpus likelihood until the target vocabulary size is reached.

Question 2

What is 'truncation' in the context of tokenizer preprocessing for transformer models?

Accepted Answer

Cutting token sequences that exceed the model's maximum input length

Answer

Truncation clips token sequences to fit within the model's maximum context window, discarding tokens beyond the limit.

Question 3

What is 'padding' in batch tokenization and why is it necessary?

Accepted Answer

Adding [PAD] tokens to shorter sequences so all sequences in a batch have the same length, enabling efficient tensor operations

Answer

Padding equalizes sequence lengths within a batch so they can be stacked into rectangular tensors required by GPU-accelerated matrix operations.

Question 4

Which of the following is a key advantage of character-level tokenization over word-level tokenization?

Accepted Answer

No out-of-vocabulary (OOV) problem since any text can be represented

Answer

Character-level tokenization can represent any string using a small fixed alphabet, completely eliminating OOV issues.

Question 5

What does 'token alignment' refer to when tokenizing text for tasks like Named Entity Recognition (NER)?

Accepted Answer

Mapping subword tokens back to their original word boundaries to correctly assign labels

Answer

In NER, word-level labels must be aligned to subword tokens; typically only the first subword of each word receives the label while others get a special ignore label.

NLP Practice Test

NLP Practice Test

NLP Tokenization 4