Natural Language Processing Text Preprocessing and Tokenization 2 — Questions and Answers
Question 1: What is a bag-of-words (BoW) representation?
- A text representation that counts word occurrences while ignoring word order (Correct answer)
- A model that predicts the next word in a sequence
- A method for extracting named entities
- A technique for measuring sentence similarity
Correct answer: A text representation that counts word occurrences while ignoring word order
Bag-of-words represents text as a multiset of words, counting their frequencies without considering grammar or order.
Question 2: What does TF-IDF stand for in NLP?
- Term Frequency–Inverse Document Frequency (Correct answer)
- Text Feature–Index Document Format
- Token Frequency–Inverted Data File
- Term Factor–Integrated Document Filter
Correct answer: Term Frequency–Inverse Document Frequency
TF-IDF weighs a term's frequency in a document against how rare it is across the corpus, boosting terms that are distinctive to a document.
Question 3: What problem does TF-IDF address that raw term frequency does not?
- It down-weights common words that appear in many documents and thus carry less discriminative power (Correct answer)
- It captures word order and syntactic structure
- It encodes semantic meaning using dense vectors
- It handles out-of-vocabulary words
Correct answer: It down-weights common words that appear in many documents and thus carry less discriminative power
TF-IDF penalizes terms that appear frequently across all documents (e.g., 'the'), preventing them from dominating feature vectors.
Question 4: Which preprocessing step involves splitting text into individual sentences?
- Sentence segmentation (Correct answer)
- Tokenization
- Lemmatization
- Chunking
Correct answer: Sentence segmentation
Sentence segmentation (or sentence boundary detection) identifies where one sentence ends and the next begins in a block of text.
Question 5: What is the purpose of padding in NLP batch processing?
- To make all sequences in a batch the same length so they can be processed together (Correct answer)
- To add special characters at the start of a sentence
- To encode positional information into embeddings
- To remove short sentences from training data
Correct answer: To make all sequences in a batch the same length so they can be processed together
Padding adds dummy tokens to shorter sequences so that all examples in a mini-batch share the same length, enabling efficient matrix operations.
Question 6: What does the special [CLS] token represent in BERT-style models?
- A classification token whose final hidden state is used as the aggregate sequence representation for classification tasks (Correct answer)
- The closing bracket of a sentence
- A token that replaces unknown words
- A separator between two sentences
Correct answer: A classification token whose final hidden state is used as the aggregate sequence representation for classification tasks
In BERT, the [CLS] token is prepended to every input, and its output embedding aggregates information from the whole sequence for downstream classification.
Natural Language Processing Text Preprocessing and Tokenization 2
What is a bag-of-words (BoW) representation?