Natural Language Processing (Artificial Intelligence) 3 — Questions and Answers
Question 1: What is the primary purpose of positional encoding in Transformer models?
- To reduce vocabulary size
- To inject sequence order information into token embeddings (Correct answer)
- To normalize attention scores
- To prevent overfitting on long sequences
Correct answer: To inject sequence order information into token embeddings
Since self-attention is permutation-invariant, positional encodings add order information to token embeddings so the model can distinguish token positions.
Question 2: Which evaluation metric computes the geometric mean of precision and recall of n-gram overlaps between a machine translation output and reference translations?
- ROUGE
- BLEU (Correct answer)
- METEOR
- BERTScore
Correct answer: BLEU
BLEU (Bilingual Evaluation Understudy) computes a modified precision over n-grams and applies a brevity penalty, then takes the geometric mean across n-gram orders.
Question 3: In transformer-based language models, what does 'masked language modeling' (MLM) train the model to do?
- Generate text autoregressively from left to right
- Predict randomly masked tokens using both left and right context (Correct answer)
- Translate sequences from one language to another
- Rank candidate sentences by fluency
Correct answer: Predict randomly masked tokens using both left and right context
MLM, used in BERT, randomly masks a portion of input tokens and trains the model to predict the original tokens using bidirectional context.
Question 4: What is 'transfer learning' in the context of NLP?
- Moving model weights between different hardware accelerators
- Using knowledge from a pre-trained model to improve performance on a new task (Correct answer)
- Translating embeddings from one language to another
- Sharing training data between competing models
Correct answer: Using knowledge from a pre-trained model to improve performance on a new task
Transfer learning leverages representations learned during pre-training on large corpora and adapts them to downstream tasks, requiring far less task-specific data.
Question 5: Which of the following best describes the difference between stemming and lemmatization?
- Stemming uses dictionaries; lemmatization uses heuristic rules
- Stemming produces morphologically valid roots; lemmatization does not
- Lemmatization returns the dictionary base form; stemming chops characters heuristically (Correct answer)
- They are identical but applied to different languages
Correct answer: Lemmatization returns the dictionary base form; stemming chops characters heuristically
Lemmatization uses vocabulary and morphological analysis to return the canonical base form, while stemming applies crude suffix-stripping rules that may not yield real words.
Question 6: What is the 'vanishing gradient' problem and which NLP architecture introduced mechanisms specifically to address it?
- Gradients become too large; addressed by CNNs
- Gradients shrink to near zero during backpropagation through long sequences; addressed by LSTMs (Correct answer)
- Loss oscillates during training; addressed by Transformers
- Embeddings collapse to zero; addressed by GloVe
Correct answer: Gradients shrink to near zero during backpropagation through long sequences; addressed by LSTMs
In deep or long sequential networks, gradients diminish exponentially during backpropagation; LSTMs introduced gating mechanisms (input, forget, output gates) to maintain gradient flow.
Question 7: In NLP, what is 'co-reference resolution'?
- Aligning parallel corpora across two languages
- Identifying when different expressions in a text refer to the same entity (Correct answer)
- Resolving ambiguity in part-of-speech tagging
- Detecting contradictions between two sentences
Correct answer: Identifying when different expressions in a text refer to the same entity
Coreference resolution links mentions like pronouns and noun phrases to the same real-world entity, which is critical for discourse understanding and information extraction.
Natural Language Processing (Artificial Intelligence) 3
What is the primary purpose of positional encoding in Transformer models?