Natural Language Processing Text Preprocessing and Tokenization 2

Question 1

What is a bag-of-words (BoW) representation?

Accepted Answer

A text representation that counts word occurrences while ignoring word order

Answer

Bag-of-words represents text as a multiset of words, counting their frequencies without considering grammar or order.

Question 2

What does TF-IDF stand for in NLP?

Accepted Answer

Term Frequency–Inverse Document Frequency

Answer

TF-IDF weighs a term's frequency in a document against how rare it is across the corpus, boosting terms that are distinctive to a document.

Question 3

What problem does TF-IDF address that raw term frequency does not?

Accepted Answer

It down-weights common words that appear in many documents and thus carry less discriminative power

Answer

TF-IDF penalizes terms that appear frequently across all documents (e.g., 'the'), preventing them from dominating feature vectors.

Question 4

Which preprocessing step involves splitting text into individual sentences?

Accepted Answer

Sentence segmentation

Answer

Sentence segmentation (or sentence boundary detection) identifies where one sentence ends and the next begins in a block of text.

Question 5

What is the purpose of padding in NLP batch processing?

Accepted Answer

To make all sequences in a batch the same length so they can be processed together

Answer

Padding adds dummy tokens to shorter sequences so that all examples in a mini-batch share the same length, enabling efficient matrix operations.

Natural Language Processing Practice Test