NLP Text Preprocessing 5 — Questions and Answers
Question 1: What is the 'vocabulary explosion' problem in NLP and how does subword tokenization address it?
- Too many stop words causing noise; subword tokenization removes them
- Unbounded vocabulary from rare/novel words; subword units cap vocabulary at a fixed size (Correct answer)
- Too many duplicate tokens; subword tokenization deduplicates them
- Excessive sentence length; subword tokenization truncates long sequences
Correct answer: Unbounded vocabulary from rare/novel words; subword units cap vocabulary at a fixed size
Subword tokenization (BPE, WordPiece) splits rare words into known subword units, enabling a fixed vocabulary to handle any new word.
Question 2: Which of the following describes the difference between type and token in corpus linguistics?
- A type is a sentence; a token is a paragraph
- A token is each individual word occurrence; a type is a unique word form (Correct answer)
- A type is a character; a token is a word
- A token is a document; a type is a corpus
Correct answer: A token is each individual word occurrence; a type is a unique word form
Tokens are all word occurrences in text (including repeats), while types are the distinct unique words; 'the cat sat on the mat' has 6 tokens but 5 types.
Question 3: In preprocessing pipeline design, why is it generally recommended to fit vectorizers only on training data?
- Test data may contain formatting differences that break the vectorizer
- Fitting on test data causes data leakage by letting the model learn test set statistics (Correct answer)
- Training data always has more vocabulary than test data
- Vectorizers cannot process unseen documents after fitting
Correct answer: Fitting on test data causes data leakage by letting the model learn test set statistics
Fitting on test data leaks information about the test distribution into the model, causing overly optimistic evaluation metrics.
Question 4: What preprocessing technique is specifically designed to handle the informal spelling variations in user-generated text (e.g., 'goooood', 'pleaseeeee')?
- Stemming
- Character repetition normalization (Correct answer)
- Named entity recognition
- Dependency parsing
Correct answer: Character repetition normalization
Character repetition normalization reduces elongated words (e.g., 'goooood' → 'good') by collapsing repeated characters to a standard form.
Question 5: How does the Punkt sentence tokenizer in NLTK handle abbreviations like 'Dr.' and 'Mr.'?
- It always treats periods after these tokens as sentence boundaries
- It learns abbreviations from training data and treats their periods as non-sentence-final (Correct answer)
- It removes all abbreviations before tokenization
- It requires manual rules for every abbreviation in a language
Correct answer: It learns abbreviations from training data and treats their periods as non-sentence-final
Punkt is an unsupervised algorithm that learns abbreviation patterns from text, avoiding false sentence splits on titles and abbreviations.
Question 6: What is the role of a 'text normalization pipeline' in an NLP system?
- To train the language model on normalized text only
- To apply a sequence of preprocessing steps transforming raw text into a consistent, clean representation (Correct answer)
- To convert text files into numerical tensors directly
- To evaluate model outputs against a gold standard
Correct answer: To apply a sequence of preprocessing steps transforming raw text into a consistent, clean representation
A text normalization pipeline chains steps like lowercasing, tokenization, stop word removal, and stemming/lemmatization into a reproducible preprocessing workflow.
Question 7: Why might preserving case information be important when preprocessing text for a Named Entity Recognition (NER) task?
- NER models require all-lowercase input for stability
- Capitalization is a strong signal for identifying proper nouns and named entities (Correct answer)
- Lowercased text reduces NER vocabulary size significantly
- NER only applies to numeric entities which are unaffected by case
Correct answer: Capitalization is a strong signal for identifying proper nouns and named entities
In English, named entities like person names, places, and organizations are typically capitalized, making case a valuable feature for NER models.
NLP Text Preprocessing 5
What is the 'vocabulary explosion' problem in NLP and how does subword tokenization address it?