NLP Text Preprocessing 2

Question 1

Which stemming algorithm is known for being the most aggressive and producing the shortest stems?

Accepted Answer

Lancaster Stemmer

Answer

The Lancaster (Paice-Husk) stemmer is the most aggressive English stemmer, often over-stemming words to very short roots.

Question 2

What does TF-IDF stand for in NLP?

Accepted Answer

Term Frequency - Inverse Document Frequency

Answer

TF-IDF stands for Term Frequency–Inverse Document Frequency, a statistical measure used to evaluate word importance in a document relative to a corpus.

Question 3

In text preprocessing, what is the purpose of removing hapax legomena?

Accepted Answer

To remove words appearing only once in the corpus

Answer

Hapax legomena are words occurring only once in a corpus; removing them reduces noise and vocabulary size without significant information loss.

Question 4

Which of the following best describes the 'bag-of-words' representation after text preprocessing?

Accepted Answer

An unordered collection of word frequencies ignoring grammar

Answer

Bag-of-words represents text as an unordered set of word counts, discarding grammar and word order information.

Question 5

What is byte pair encoding (BPE) primarily used for in text preprocessing?

Accepted Answer

Subword tokenization to handle out-of-vocabulary words

Answer

BPE iteratively merges frequent character pairs to build a subword vocabulary, enabling models to handle rare and unknown words.

NLP Practice Test

NLP Practice Test

NLP Text Preprocessing 2