NLP Text Preprocessing 2 — Questions and Answers
Question 1: Which stemming algorithm is known for being the most aggressive and producing the shortest stems?
- Porter Stemmer
- Snowball Stemmer
- Lancaster Stemmer (Correct answer)
- Lovins Stemmer
Correct answer: Lancaster Stemmer
The Lancaster (Paice-Husk) stemmer is the most aggressive English stemmer, often over-stemming words to very short roots.
Question 2: What does TF-IDF stand for in NLP?
- Text Frequency - Inverse Document Format
- Term Frequency - Inverse Document Frequency (Correct answer)
- Token Frequency - Internal Document Filter
- Text Filtering - Index Document Feature
Correct answer: Term Frequency - Inverse Document Frequency
TF-IDF stands for Term Frequency–Inverse Document Frequency, a statistical measure used to evaluate word importance in a document relative to a corpus.
Question 3: In text preprocessing, what is the purpose of removing hapax legomena?
- To eliminate duplicate sentences
- To remove words appearing only once in the corpus (Correct answer)
- To strip HTML tags from text
- To normalize Unicode characters
Correct answer: To remove words appearing only once in the corpus
Hapax legomena are words occurring only once in a corpus; removing them reduces noise and vocabulary size without significant information loss.
Question 4: Which of the following best describes the 'bag-of-words' representation after text preprocessing?
- A sequential model preserving word order
- An unordered collection of word frequencies ignoring grammar (Correct answer)
- A hierarchical tree of syntactic phrases
- A graph of co-occurring word pairs
Correct answer: An unordered collection of word frequencies ignoring grammar
Bag-of-words represents text as an unordered set of word counts, discarding grammar and word order information.
Question 5: What is byte pair encoding (BPE) primarily used for in text preprocessing?
- Compressing stop word lists
- Subword tokenization to handle out-of-vocabulary words (Correct answer)
- Encoding text as binary for storage
- Normalizing punctuation characters
Correct answer: Subword tokenization to handle out-of-vocabulary words
BPE iteratively merges frequent character pairs to build a subword vocabulary, enabling models to handle rare and unknown words.
Question 6: When lowercasing text, which scenario presents the greatest risk of information loss?
- Converting 'Apple' to 'apple'
- Converting named entity 'US' to 'us' (Correct answer)
- Converting 'The' to 'the'
- Converting 'NLP' to 'nlp'
Correct answer: Converting named entity 'US' to 'us'
Converting 'US' (United States) to 'us' (pronoun) conflates two entirely different meanings, making disambiguation impossible downstream.
Question 7: What is the main advantage of using a lemmatizer over a stemmer?
- Lemmatizers are always faster to compute
- Lemmatizers return valid dictionary words based on morphological analysis (Correct answer)
- Lemmatizers require no external vocabulary
- Lemmatizers handle all languages identically
Correct answer: Lemmatizers return valid dictionary words based on morphological analysis
Lemmatizers use morphological analysis and a dictionary to return actual base forms (lemmas), while stemmers use heuristic rules that may produce non-words.
NLP Text Preprocessing 2
Which stemming algorithm is known for being the most aggressive and producing the shortest stems?