Artificial Intelligence Natural Language Processing 1 — Questions and Answers
Question 1: What is tokenization in natural language processing?
- Converting text to uppercase
- Splitting text into meaningful units such as words or subwords (Correct answer)
- Measuring the sentiment of a sentence
- Removing stop words from a document
Correct answer: Splitting text into meaningful units such as words or subwords
Tokenization breaks raw text into tokens (words, subwords, or characters) that serve as the basic units for NLP model input.
Question 2: What does TF-IDF measure in information retrieval and NLP?
- The grammatical structure of a sentence
- The importance of a word in a document relative to a corpus (Correct answer)
- The translation probability between two languages
- The semantic similarity between two documents
Correct answer: The importance of a word in a document relative to a corpus
TF-IDF multiplies term frequency (how often a word appears in a document) by inverse document frequency (penalizing words common across many documents).
Question 3: Which word embedding model learns vector representations by predicting surrounding words (skip-gram) or predicting a word from context (CBOW)?
- GloVe
- BERT
- Word2Vec (Correct answer)
- FastText
Correct answer: Word2Vec
Word2Vec offers two architectures: skip-gram predicts context words from a target, and CBOW predicts a target word from its context window.
Question 4: What NLP task involves labeling each token in a sentence with its grammatical role (noun, verb, etc.)?
- Named entity recognition
- Sentiment analysis
- Part-of-speech tagging (Correct answer)
- Dependency parsing
Correct answer: Part-of-speech tagging
Part-of-speech (POS) tagging assigns grammatical categories to each token, such as noun, verb, adjective, or adverb.
Question 5: What is the 'attention mechanism' in neural NLP models?
- A technique for removing irrelevant tokens
- A method that allows the model to weight the importance of different input tokens when producing an output (Correct answer)
- A way to speed up tokenization
- A rule for choosing the correct part of speech
Correct answer: A method that allows the model to weight the importance of different input tokens when producing an output
Attention lets the model dynamically focus on relevant parts of the input sequence when generating each output token, enabling better handling of long-range dependencies.
Question 6: What is named entity recognition (NER) in NLP?
- Detecting the language of a text
- Identifying and classifying real-world entities such as people, organizations, and locations in text (Correct answer)
- Counting the frequency of words in a corpus
- Generating summaries of long documents
Correct answer: Identifying and classifying real-world entities such as people, organizations, and locations in text
NER identifies spans of text that refer to specific entity categories (persons, organizations, dates, etc.) and tags them accordingly.
Artificial Intelligence Natural Language Processing 1
What is tokenization in natural language processing?