NLP Text Preprocessing 5

Question 1

What is the 'vocabulary explosion' problem in NLP and how does subword tokenization address it?

Accepted Answer

Unbounded vocabulary from rare/novel words; subword units cap vocabulary at a fixed size

Answer

Subword tokenization (BPE, WordPiece) splits rare words into known subword units, enabling a fixed vocabulary to handle any new word.

Question 2

Which of the following describes the difference between type and token in corpus linguistics?

Accepted Answer

A token is each individual word occurrence; a type is a unique word form

Answer

Tokens are all word occurrences in text (including repeats), while types are the distinct unique words; 'the cat sat on the mat' has 6 tokens but 5 types.

Question 3

In preprocessing pipeline design, why is it generally recommended to fit vectorizers only on training data?

Accepted Answer

Fitting on test data causes data leakage by letting the model learn test set statistics

Answer

Fitting on test data leaks information about the test distribution into the model, causing overly optimistic evaluation metrics.

Question 4

What preprocessing technique is specifically designed to handle the informal spelling variations in user-generated text (e.g., 'goooood', 'pleaseeeee')?

Accepted Answer

Character repetition normalization

Answer

Character repetition normalization reduces elongated words (e.g., 'goooood' → 'good') by collapsing repeated characters to a standard form.

Question 5

How does the Punkt sentence tokenizer in NLTK handle abbreviations like 'Dr.' and 'Mr.'?

Accepted Answer

It learns abbreviations from training data and treats their periods as non-sentence-final

Answer

Punkt is an unsupervised algorithm that learns abbreviation patterns from text, avoiding false sentence splits on titles and abbreviations.

NLP Practice Test

NLP Practice Test

NLP Text Preprocessing 5