NLP Word Embeddings 2 — Questions and Answers
Question 1: Which property of word2vec embeddings allows the analogy 'king - man + woman ≈ queen' to work?
- Linear relationship encoding of semantic roles (Correct answer)
- Random initialization of vectors
- Softmax normalization of output layer
- Bag-of-words context window
Correct answer: Linear relationship encoding of semantic roles
Word2vec encodes semantic relationships as linear offsets in the embedding space, enabling arithmetic analogies.
Question 2: What is the primary difference between the Skip-gram and CBOW architectures in word2vec?
- Skip-gram predicts context from a target word; CBOW predicts a target from context words (Correct answer)
- Skip-gram uses larger vocabularies; CBOW uses smaller ones
- Skip-gram is faster to train; CBOW is slower
- Skip-gram uses character-level inputs; CBOW uses word-level inputs
Correct answer: Skip-gram predicts context from a target word; CBOW predicts a target from context words
Skip-gram takes a center word and predicts surrounding context words, while CBOW averages context words to predict the center word.
Question 3: What does 'negative sampling' accomplish in word2vec training?
- Approximates the softmax by training on a small set of noise words alongside the target (Correct answer)
- Removes words with negative sentiment from the vocabulary
- Samples words with below-average frequency for training
- Discards embeddings with negative vector components
Correct answer: Approximates the softmax by training on a small set of noise words alongside the target
Negative sampling makes training tractable by updating weights only for the target word and a small random sample of non-target words.
Question 4: In GloVe (Global Vectors), what is the main training objective?
- Factorize the global word co-occurrence matrix into low-rank embeddings (Correct answer)
- Maximize next-word prediction probability using a neural language model
- Minimize reconstruction error in a denoising autoencoder
- Classify word pairs as synonyms or antonyms
Correct answer: Factorize the global word co-occurrence matrix into low-rank embeddings
GloVe directly factorizes the log of the global co-occurrence count matrix to produce word vectors.
Question 5: Which evaluation method tests word embeddings by checking whether cosine similarity rankings match human-rated similarity scores?
- Intrinsic evaluation via word similarity benchmarks (Correct answer)
- Extrinsic evaluation on downstream NLP tasks
- Perplexity measurement on a held-out corpus
- Cross-entropy scoring against a language model
Correct answer: Intrinsic evaluation via word similarity benchmarks
Intrinsic evaluation compares embedding-derived similarity rankings against human-annotated datasets like WordSim-353.
Question 6: Why do subword-based embeddings like fastText outperform word2vec on morphologically rich languages?
- They represent words as sums of character n-gram vectors, handling unseen word forms (Correct answer)
- They use a larger corpus for training
- They apply attention over the full sentence context
- They rely on pre-trained transformer layers
Correct answer: They represent words as sums of character n-gram vectors, handling unseen word forms
FastText decomposes each word into overlapping character n-grams, so it can construct embeddings for out-of-vocabulary words with shared morphemes.
Question 7: What does 'embedding dimensionality' control in a word embedding model?
- The size of the dense vector representing each word, balancing expressiveness and memory (Correct answer)
- The number of words in the vocabulary
- The length of the context window used during training
- The number of training epochs
Correct answer: The size of the dense vector representing each word, balancing expressiveness and memory
Dimensionality determines how many features each word vector has; higher dimensions can capture more nuance but require more memory and data.
NLP Word Embeddings 2
Which property of word2vec embeddings allows the analogy 'king - man + woman ≈ queen' to work?