MS-DS Master of Data science Natural Language Processing 3 — Questions and Answers
Question 1: In transformer-based models, what is the role of the feed-forward sublayer that follows multi-head attention in each encoder block?
- To compute cross-attention with the decoder
- To apply a position-wise non-linear transformation independently to each token (Correct answer)
- To reduce sequence length via pooling
- To convert token embeddings to one-hot vectors
Correct answer: To apply a position-wise non-linear transformation independently to each token
The position-wise feed-forward network applies two linear transformations with a ReLU (or GELU) activation to each token representation independently, adding non-linearity.
Question 2: What problem does label smoothing address during training of NLP classifiers?
- Class imbalance in the training set
- Overconfident softmax predictions that hurt generalization (Correct answer)
- Vanishing gradients in deep RNNs
- Out-of-vocabulary tokens in the decoder
Correct answer: Overconfident softmax predictions that hurt generalization
Label smoothing replaces hard 0/1 targets with soft distributions (e.g., 0.9 / 0.1/n), preventing the model from becoming overconfident and improving calibration.
Question 3: Which technique is used in ELMo to create context-sensitive word representations?
- Static GloVe vectors averaged across senses
- Bidirectional LSTM language models whose hidden states are combined (Correct answer)
- Single-direction Transformer with positional encoding
- Sparse random projections of co-occurrence matrices
Correct answer: Bidirectional LSTM language models whose hidden states are combined
ELMo (Embeddings from Language Models) uses a two-layer bidirectional LSTM trained as a language model; representations are task-specific linear combinations of all layer states.
Question 4: In dependency parsing, what does a 'head' and 'dependent' relationship capture?
- The frequency of co-occurrence between two words
- A directed binary grammatical relation between a governing word and a word it governs (Correct answer)
- The sequential distance between two tokens
- The shared semantic role of two arguments
Correct answer: A directed binary grammatical relation between a governing word and a word it governs
In dependency grammar, each word (except the root) has exactly one head; the arc direction and label (e.g., nsubj, dobj) encode the syntactic relation.
Question 5: What is the vanishing gradient problem in the context of training recurrent neural networks on long sequences?
- Gradients become very large, causing weight explosion
- Gradients shrink exponentially through time steps, preventing learning of long-range dependencies (Correct answer)
- The loss function becomes non-convex for sequences longer than 50 tokens
- Attention weights collapse to uniform distribution over time
Correct answer: Gradients shrink exponentially through time steps, preventing learning of long-range dependencies
When backpropagating through many time steps, repeated multiplication of small Jacobians causes gradients to vanish, making it difficult to capture distant dependencies.
Question 6: Which of the following best describes the concept of 'zero-shot' learning in NLP?
- Training a model with zero labeled examples overall
- Applying a pre-trained model to a task it was never explicitly fine-tuned on (Correct answer)
- Using zero regularization during fine-tuning
- Evaluating a model on a held-out test set with zero overlap with training classes
Correct answer: Applying a pre-trained model to a task it was never explicitly fine-tuned on
Zero-shot learning leverages a model's pre-trained knowledge and natural language task descriptions to perform tasks without any task-specific training examples.
Question 7: In the Transformer decoder, why is masked self-attention used during training?
- To reduce computational cost by attending only to nearby tokens
- To prevent the model from attending to future tokens, preserving autoregressive generation (Correct answer)
- To mask padding tokens in variable-length batches
- To block cross-attention from the encoder during the first decoder layer
Correct answer: To prevent the model from attending to future tokens, preserving autoregressive generation
During training, future token positions are masked with -∞ before softmax so the model cannot 'cheat' by looking ahead, mirroring inference-time generation.
MS-DS Master of Data science Natural Language Processing 3
In transformer-based models, what is the role of the feed-forward sublayer that follows multi-head attention in each encoder block?