Data Science with Python Certification Data Science with Python Natural Language Processing 1 — Questions and Answers
Question 1: Which Python library provides the `word_tokenize()` function for splitting text into individual tokens?
- spaCy
- NLTK (Correct answer)
- TextBlob
- Gensim
Correct answer: NLTK
NLTK (Natural Language Toolkit) provides `word_tokenize()`, which splits a string into a list of word and punctuation tokens.
Question 2: What is the primary purpose of removing stop words in NLP preprocessing?
- To reduce vocabulary size and eliminate low-information words (Correct answer)
- To correct spelling errors in the text
- To convert words to their base grammatical form
- To split sentences into individual characters
Correct answer: To reduce vocabulary size and eliminate low-information words
Stop words like 'the', 'is', and 'and' carry little semantic meaning; removing them reduces noise and shrinks the feature space.
Question 3: What is the difference between stemming and lemmatization in NLP?
- Stemming uses a vocabulary lookup while lemmatization applies fixed suffix rules
- Stemming crudely chops suffixes while lemmatization returns a valid dictionary base form (Correct answer)
- Stemming is more accurate than lemmatization in all cases
- They are functionally identical processes with different library implementations
Correct answer: Stemming crudely chops suffixes while lemmatization returns a valid dictionary base form
Stemming applies heuristic suffix stripping (e.g., 'running' → 'run'), while lemmatization uses morphological analysis to return the proper dictionary form.
Question 4: In Python's NLTK library, which class implements Porter stemming?
- nltk.stem.WordNetLemmatizer
- nltk.stem.PorterStemmer (Correct answer)
- nltk.stem.SnowballStemmer
- nltk.tokenize.RegexpTokenizer
Correct answer: nltk.stem.PorterStemmer
`nltk.stem.PorterStemmer` implements the Porter algorithm, one of the oldest and most commonly used stemming algorithms.
Question 5: What does a 'corpus' refer to in the context of NLP?
- A single tokenized sentence
- A large structured collection of text used for training or analysis (Correct answer)
- The vocabulary dictionary of a language model
- A confusion matrix for text classification
Correct answer: A large structured collection of text used for training or analysis
A corpus is a large, structured collection of text documents used to train models, build vocabulary, or perform linguistic analysis.
Question 6: Which of the following best describes a 'bag of words' (BoW) representation?
- A sequence model that preserves word order and context
- A word embedding that maps words to dense vectors
- A document representation as an unordered set of word frequency counts (Correct answer)
- A parse tree showing grammatical structure
Correct answer: A document representation as an unordered set of word frequency counts
BoW represents a document as a vector of word occurrence counts, discarding grammar and word order entirely.
Question 7: What does TF-IDF measure in text analysis?
- The total frequency of all terms across all documents
- The importance of a word to a document relative to a collection of documents (Correct answer)
- The average sentence length across a corpus
- The cosine similarity between two document vectors
Correct answer: The importance of a word to a document relative to a collection of documents
TF-IDF (Term Frequency–Inverse Document Frequency) scores a term by how often it appears in a document (TF) discounted by how common it is across all documents (IDF), highlighting distinctive words.
Data Science with Python Certification Data Science with Python Natural Language Processing 1
Which Python library provides the `word_tokenize()` function for splitting text into individual tokens?