Natural Language Processing Language Models and Neural NLP 1 — Questions and Answers
Question 1: What is a word embedding in NLP?
- A dense, low-dimensional vector representation of a word that captures semantic meaning (Correct answer)
- A binary one-hot encoding of words
- A list of synonyms for a word
- A frequency count of a word in a corpus
Correct answer: A dense, low-dimensional vector representation of a word that captures semantic meaning
Word embeddings map words to dense real-valued vectors in a continuous space where semantically similar words are geometrically close.
Question 2: What training objective does Word2Vec's Skip-gram model use?
- Predict surrounding context words given a center word (Correct answer)
- Predict the center word given surrounding context words
- Reconstruct the input sequence from a compressed representation
- Classify a sentence as positive or negative
Correct answer: Predict surrounding context words given a center word
Skip-gram trains a model to predict the words surrounding a given target word, learning embeddings that encode co-occurrence patterns.
Question 3: What does the attention mechanism in transformers compute?
- A weighted sum of value vectors based on the relevance of each key to the query (Correct answer)
- The probability distribution over the next token in a sequence
- The gradient of the loss with respect to each weight
- A bag-of-words representation of the input
Correct answer: A weighted sum of value vectors based on the relevance of each key to the query
Attention computes compatibility scores between a query and all keys, uses softmax to get weights, and produces a weighted sum of corresponding values.
Question 4: What is the primary innovation of the Transformer architecture over RNNs?
- It uses self-attention to process all tokens in parallel rather than sequentially (Correct answer)
- It uses larger embedding dimensions
- It applies convolutions to text instead of recurrence
- It trains only on labeled data
Correct answer: It uses self-attention to process all tokens in parallel rather than sequentially
Transformers process the entire sequence simultaneously via self-attention, enabling parallelization and capturing long-range dependencies without vanishing gradients.
Question 5: What does BERT's masked language modeling (MLM) pre-training objective do?
- Randomly masks tokens in the input and trains the model to predict the masked words (Correct answer)
- Predicts the next sentence given the current sentence
- Generates text from left to right like a language model
- Classifies whether two sentences are paraphrases
Correct answer: Randomly masks tokens in the input and trains the model to predict the masked words
MLM masks 15% of input tokens and trains BERT to reconstruct them, enabling bidirectional context learning.
Question 6: What distinguishes GPT-style models from BERT-style models?
- GPT uses a left-to-right causal (decoder-only) language model, while BERT uses a bidirectional encoder (Correct answer)
- GPT is smaller than BERT
- BERT generates text while GPT classifies it
- GPT does not use the transformer architecture
Correct answer: GPT uses a left-to-right causal (decoder-only) language model, while BERT uses a bidirectional encoder
GPT is an autoregressive decoder that sees only left context, making it suitable for text generation, whereas BERT is a bidirectional encoder suited for understanding tasks.
Natural Language Processing Language Models and Neural NLP 1
What is a word embedding in NLP?