Natural Language Processing Language Models and Neural NLP 2 — Questions and Answers
Question 1: What is transfer learning in the context of NLP?
- Pre-training a large model on general text data then fine-tuning it on a specific downstream task (Correct answer)
- Transferring model weights from one programming framework to another
- Using a model trained on images to process text
- Copying training data from one dataset to another
Correct answer: Pre-training a large model on general text data then fine-tuning it on a specific downstream task
Transfer learning allows NLP models to leverage knowledge from large-scale pre-training and adapt it to task-specific data with minimal additional training.
Question 2: What is the vanishing gradient problem in RNNs?
- Gradients become extremely small during backpropagation through long sequences, preventing the model from learning long-range dependencies (Correct answer)
- The model generates repetitive text
- Weights grow unboundedly during training
- The model fails to tokenize long sentences
Correct answer: Gradients become extremely small during backpropagation through long sequences, preventing the model from learning long-range dependencies
In deep or long-sequence RNNs, gradients shrink exponentially as they propagate back through time steps, making it hard to learn dependencies between distant tokens.
Question 3: What gates does an LSTM unit use to control information flow?
- Input gate, forget gate, and output gate (Correct answer)
- Attention gate, reset gate, and update gate
- Encode gate, decode gate, and memory gate
- Forward gate, backward gate, and neutral gate
Correct answer: Input gate, forget gate, and output gate
LSTMs use three gates—input (what new info to store), forget (what to discard from cell state), and output (what to expose as the hidden state)—to regulate long-term memory.
Question 4: What is positional encoding used for in transformer models?
- To inject information about each token's position in the sequence since self-attention is permutation-invariant (Correct answer)
- To normalize the embedding vectors
- To map tokens to continuous vectors
- To compute attention scores between tokens
Correct answer: To inject information about each token's position in the sequence since self-attention is permutation-invariant
Because self-attention treats the sequence as a set, positional encodings add order information so the model knows which token comes first, second, etc.
Question 5: What is perplexity as a metric for language models?
- The exponentiated average negative log-likelihood of a test corpus; lower perplexity means better prediction (Correct answer)
- The percentage of tokens the model classifies incorrectly
- The number of parameters in the model
- The average sentence length in the training corpus
Correct answer: The exponentiated average negative log-likelihood of a test corpus; lower perplexity means better prediction
Perplexity measures how surprised a language model is by unseen text — a perplexity of N means the model is as confused as if it had to choose uniformly among N options at each step.
Question 6: What is the role of the softmax function at the output layer of a language model?
- To convert raw logit scores into a probability distribution over the vocabulary (Correct answer)
- To normalize the input embeddings
- To apply dropout regularization
- To compute cross-entropy loss
Correct answer: To convert raw logit scores into a probability distribution over the vocabulary
Softmax exponentiates each logit and divides by the sum, ensuring all vocabulary probabilities are positive and sum to 1.
Natural Language Processing Language Models and Neural NLP 2
What is transfer learning in the context of NLP?