NLP Tokenization 2

Question 1

What is the primary purpose of Byte Pair Encoding (BPE) in tokenization?

Accepted Answer

To iteratively merge the most frequent adjacent byte pairs into a single token

Answer

BPE starts with individual characters and iteratively merges the most frequent adjacent pairs to build a vocabulary of subword units.

Question 2

Which tokenization strategy is best suited for handling out-of-vocabulary (OOV) words?

Accepted Answer

Subword tokenization

Answer

Subword tokenization breaks unknown words into known subword pieces, effectively eliminating the OOV problem.

Question 3

In the context of tokenization, what does the term 'vocabulary size' refer to?

Accepted Answer

The total number of unique tokens the tokenizer can produce

Answer

Vocabulary size is the count of distinct tokens (words, subwords, or characters) that a tokenizer's model recognizes.

Question 4

Which tokenizer is used by the original BERT model?

Accepted Answer

WordPiece

Answer

BERT uses WordPiece tokenization, which builds a vocabulary by merging token pairs that maximize the likelihood of the training data.

Question 5

What distinguishes WordPiece from BPE in subword tokenization?

Accepted Answer

BPE merges based on frequency, WordPiece merges based on likelihood of the training data

Answer

BPE selects the most frequent pair to merge, while WordPiece selects the pair whose merge maximizes the training corpus likelihood.

NLP Practice Test