Natural Language Processing Text Preprocessing and Tokenization 1 — Questions and Answers
Question 1: What is tokenization in NLP?
- Splitting text into individual units such as words or subwords (Correct answer)
- Removing stop words from a sentence
- Converting text to lowercase
- Stemming words to their root form
Correct answer: Splitting text into individual units such as words or subwords
Tokenization is the process of splitting raw text into smaller units called tokens, which can be words, subwords, or characters.
Question 2: Which of the following is a common stop word in English NLP preprocessing?
- the (Correct answer)
- neural
- embedding
- corpus
Correct answer: the
Stop words like 'the', 'is', and 'a' are frequently removed during preprocessing because they carry little semantic meaning.
Question 3: What does stemming do in NLP?
- Reduces a word to its base or root form by removing suffixes (Correct answer)
- Converts words into numerical vectors
- Identifies named entities in text
- Removes punctuation from text
Correct answer: Reduces a word to its base or root form by removing suffixes
Stemming strips affixes from words to produce a common base form, e.g., 'running' and 'runs' both stem to 'run'.
Question 4: How does lemmatization differ from stemming?
- Lemmatization returns the dictionary base form using vocabulary analysis, while stemming uses heuristic rules (Correct answer)
- Stemming is more accurate than lemmatization
- Lemmatization removes stop words, stemming does not
- Stemming uses a dictionary, lemmatization uses rules
Correct answer: Lemmatization returns the dictionary base form using vocabulary analysis, while stemming uses heuristic rules
Lemmatization uses morphological analysis and a dictionary to return the true base form of a word, making it more linguistically accurate than stemming.
Question 5: What is the purpose of lowercasing text during NLP preprocessing?
- To reduce vocabulary size by treating 'Apple' and 'apple' as the same token (Correct answer)
- To improve model speed by reducing character count
- To remove named entities from the text
- To convert digits into words
Correct answer: To reduce vocabulary size by treating 'Apple' and 'apple' as the same token
Lowercasing normalizes text so that words with different capitalizations are treated identically, reducing vocabulary size.
Question 6: What is Byte-Pair Encoding (BPE) used for in NLP?
- Subword tokenization to handle rare and out-of-vocabulary words (Correct answer)
- Compressing model weights to reduce memory
- Encoding text as binary for storage
- Splitting paragraphs into sentences
Correct answer: Subword tokenization to handle rare and out-of-vocabulary words
BPE is a subword tokenization algorithm that iteratively merges frequent character pairs, enabling models to handle unseen words by breaking them into known subunits.
Natural Language Processing Text Preprocessing and Tokenization 1
What is tokenization in NLP?