Natural Language Processing (Artificial Intelligence) 4 — Questions and Answers
Question 1: Which technique reduces the dimensionality of word embeddings while preserving semantic relationships, commonly used before the deep learning era?
- Byte-pair encoding
- Latent Semantic Analysis (LSA) (Correct answer)
- Conditional random fields
- CYK parsing
Correct answer: Latent Semantic Analysis (LSA)
LSA applies SVD to a term-document matrix to discover latent semantic structure and reduce dimensionality, capturing synonymy and related concepts.
Question 2: What is 'instruction tuning' in large language models?
- Training exclusively on code to follow syntax rules
- Fine-tuning a model on examples of (instruction, desired output) pairs to better follow natural language directives (Correct answer)
- Increasing model depth to handle longer instructions
- Teaching the model to refuse harmful prompts
Correct answer: Fine-tuning a model on examples of (instruction, desired output) pairs to better follow natural language directives
Instruction tuning fine-tunes a pre-trained LLM on diverse (instruction, response) pairs, dramatically improving the model's ability to follow zero-shot user commands.
Question 3: In text classification, what does 'TF-IDF' weighting accomplish?
- It normalizes sentence length to prevent bias toward longer documents
- It up-weights terms that are frequent in a document but rare across the corpus (Correct answer)
- It converts text to fixed-length embeddings using neural networks
- It assigns POS tags to each token before classification
Correct answer: It up-weights terms that are frequent in a document but rare across the corpus
TF-IDF balances term frequency (local importance) against inverse document frequency (global rarity), surfacing distinctive words while down-weighting common stopwords.
Question 4: What distinguishes a 'generative' language model from a 'discriminative' model in NLP?
- Generative models only work on classification; discriminative models handle generation
- Generative models learn the joint probability P(x,y); discriminative models learn P(y|x) (Correct answer)
- Generative models require labeled data; discriminative models do not
- Discriminative models use attention; generative models use recurrence
Correct answer: Generative models learn the joint probability P(x,y); discriminative models learn P(y|x)
Generative models model the full joint distribution and can sample new data, while discriminative models focus only on the conditional distribution needed for prediction.
Question 5: Which decoding strategy for language generation introduces randomness by sampling from the top-k most probable next tokens?
- Greedy decoding
- Beam search
- Top-k sampling (Correct answer)
- Nucleus (top-p) sampling
Correct answer: Top-k sampling
Top-k sampling restricts sampling to only the k highest-probability tokens at each step, balancing diversity and coherence in generated text.
Question 6: What is 'dependency parsing' in NLP?
- Grouping tokens into non-overlapping spans such as noun phrases and verb phrases
- Identifying binary grammatical relationships between pairs of words in a sentence (Correct answer)
- Segmenting a document into topically coherent sections
- Resolving which sense of a polysemous word is intended
Correct answer: Identifying binary grammatical relationships between pairs of words in a sentence
Dependency parsing constructs a directed graph where edges represent grammatical relations (subject, object, modifier) between a head word and its dependents.
Question 7: In neural machine translation, what is the purpose of the 'attention mechanism' introduced before Transformers (Bahdanau attention)?
- To replace the recurrent encoder with a feedforward network
- To allow the decoder to dynamically focus on relevant encoder hidden states when generating each target token (Correct answer)
- To reduce memory usage during training on long sequences
- To align source and target vocabularies during data preprocessing
Correct answer: To allow the decoder to dynamically focus on relevant encoder hidden states when generating each target token
Bahdanau attention lets the decoder compute a context vector as a weighted sum of all encoder states, solving the bottleneck of compressing source sentences into a single fixed vector.
Natural Language Processing (Artificial Intelligence) 4
Which technique reduces the dimensionality of word embeddings while preserving semantic relationships, commonly used before the deep learning era?