Natural Language Processing Text Preprocessing and Tokenization

Question 1

What is tokenization in NLP?

Accepted Answer

Splitting text into individual units such as words or subwords

Answer

Tokenization is the process of splitting raw text into smaller units called tokens, which can be words, subwords, or characters.

Question 2

Which of the following is a common stop word in English NLP preprocessing?

Accepted Answer

the

Answer

Stop words like 'the', 'is', and 'a' are frequently removed during preprocessing because they carry little semantic meaning.

Question 3

What does stemming do in NLP?

Accepted Answer

Reduces a word to its base or root form by removing suffixes

Answer

Stemming strips affixes from words to produce a common base form, e.g., 'running' and 'runs' both stem to 'run'.

Question 4

How does lemmatization differ from stemming?

Accepted Answer

Lemmatization returns the dictionary base form using vocabulary analysis, while stemming uses heuristic rules

Answer

Lemmatization uses morphological analysis and a dictionary to return the true base form of a word, making it more linguistically accurate than stemming.

Question 5

What is the purpose of lowercasing text during NLP preprocessing?

Accepted Answer

To reduce vocabulary size by treating 'Apple' and 'apple' as the same token

Answer

Lowercasing normalizes text so that words with different capitalizations are treated identically, reducing vocabulary size.

Natural Language Processing Practice Test

Natural Language Processing Practice Test

Natural Language Processing Text Preprocessing and Tokenization