Natural Language Processing Text Preprocessing and Tokenization

Question 1

What is tokenization in NLP?

Accepted Answer

Splitting text into individual units such as words or subwords

Answer

Removing stop words from a sentence

Answer

Converting text to lowercase

Answer

Stemming words to their root form

Question 2

Which of the following is a common stop word in English NLP preprocessing?

Accepted Answer

the

Answer

neural

Answer

embedding

Answer

corpus

Question 3

What does stemming do in NLP?

Accepted Answer

Reduces a word to its base or root form by removing suffixes

Answer

Converts words into numerical vectors

Answer

Identifies named entities in text

Answer

Removes punctuation from text

Question 4

How does lemmatization differ from stemming?

Accepted Answer

Lemmatization returns the dictionary base form using vocabulary analysis, while stemming uses heuristic rules

Answer

Stemming is more accurate than lemmatization

Answer

Lemmatization removes stop words, stemming does not

Answer

Stemming uses a dictionary, lemmatization uses rules

Question 5

What is the purpose of lowercasing text during NLP preprocessing?

Accepted Answer

To reduce vocabulary size by treating 'Apple' and 'apple' as the same token

Answer

To improve model speed by reducing character count

Answer

To remove named entities from the text

Answer

To convert digits into words

Question 6

What is Byte-Pair Encoding (BPE) used for in NLP?

Accepted Answer

Subword tokenization to handle rare and out-of-vocabulary words

Answer

Compressing model weights to reduce memory

Answer

Encoding text as binary for storage

Answer

Splitting paragraphs into sentences