NLP Text Preprocessing 4 — Questions and Answers
Question 1: What preprocessing challenge arises specifically when handling Chinese or Japanese text that doesn't exist with English?
- Capitalization normalization
- Word boundary segmentation since words are not space-delimited (Correct answer)
- Removing punctuation marks
- Handling contractions
Correct answer: Word boundary segmentation since words are not space-delimited
Chinese and Japanese text lacks spaces between words, so word segmentation algorithms are required before any word-level processing.
Question 2: What is the effect of applying log normalization to term frequency (TF) values?
- It makes all TF values equal to 1
- It reduces the impact of very high frequency terms by compressing the scale (Correct answer)
- It converts TF values to binary present/absent flags
- It increases the weight of rare terms exponentially
Correct answer: It reduces the impact of very high frequency terms by compressing the scale
Log normalization (e.g., 1 + log(tf)) prevents very frequent terms from dominating by compressing large frequency differences.
Question 3: In preprocessing a legal or medical corpus, why might domain-specific stop words be added beyond the standard list?
- Legal text has no stop words since every word is important
- Common domain terms like 'whereas' or 'patient' appear frequently but carry little discriminative value (Correct answer)
- Medical text uses only numeric tokens
- Domain-specific words are always meaningful and should never be removed
Correct answer: Common domain terms like 'whereas' or 'patient' appear frequently but carry little discriminative value
High-frequency domain terms that appear in nearly every document (like 'patient' in medical text) behave like stop words and can be added to a custom list.
Question 4: What is a 'character n-gram' and why is it useful in text preprocessing?
- A sequence of n words used for language modeling
- A sequence of n consecutive characters, useful for handling misspellings and morphology (Correct answer)
- A byte-level encoding of Unicode text
- A fixed-length hash of a text window
Correct answer: A sequence of n consecutive characters, useful for handling misspellings and morphology
Character n-grams capture subword patterns, making models robust to typos, morphological variants, and out-of-vocabulary words.
Question 5: Which text normalization step would convert '5 kilometers' and '5 km' to a consistent representation?
- Stemming
- Stop word removal
- Unit normalization or entity normalization (Correct answer)
- Sentence segmentation
Correct answer: Unit normalization or entity normalization
Unit normalization maps different representations of the same measurement (km, kilometers, kilometre) to a canonical form.
Question 6: When preprocessing text for a sentiment analysis model, removing which of the following would likely hurt performance the most?
- HTML tags
- Negation words like 'not', 'never', 'hardly' (Correct answer)
- Boilerplate website footers
- Repeated punctuation like '...'
Correct answer: Negation words like 'not', 'never', 'hardly'
Negation words fundamentally flip sentiment polarity (e.g., 'not good' = negative), so removing them severely degrades sentiment classification.
Question 7: What is the primary purpose of sentence boundary detection (SBD) in text preprocessing?
- To identify the grammatical subject of each sentence
- To split a document into individual sentences for downstream processing (Correct answer)
- To detect incomplete or malformed sentences
- To label sentences with their discourse function
Correct answer: To split a document into individual sentences for downstream processing
SBD (also called sentence segmentation) correctly identifies where one sentence ends and another begins, enabling sentence-level analysis.
NLP Text Preprocessing 4
What preprocessing challenge arises specifically when handling Chinese or Japanese text that doesn't exist with English?