NLP Word Embeddings 3 — Questions and Answers
Question 1: Which phenomenon explains why 'bank' has different meanings in 'river bank' versus 'savings bank,' and how do static embeddings handle it?
- Polysemy; static embeddings assign one fixed vector per word regardless of context (Correct answer)
- Synonymy; static embeddings merge both senses into a zero vector
- Homography; static embeddings duplicate the word in the vocabulary
- Metonymy; static embeddings ignore punctuation around the word
Correct answer: Polysemy; static embeddings assign one fixed vector per word regardless of context
Static embeddings like word2vec average all senses of a polysemous word into a single vector, losing context-specific meaning.
Question 2: What role does the projection layer play in a word2vec neural network?
- It maps one-hot input vectors to dense lower-dimensional embedding vectors (Correct answer)
- It applies non-linear activation to hidden representations
- It computes attention weights across the sequence
- It generates probability distributions over the vocabulary using a recurrent unit
Correct answer: It maps one-hot input vectors to dense lower-dimensional embedding vectors
The projection layer is a simple linear lookup that converts sparse one-hot word representations into dense embedding vectors without a non-linearity.
Question 3: What is 'subsampling of frequent words' in word2vec, and why is it used?
- Randomly discarding high-frequency words during training to reduce noise and speed training (Correct answer)
- Downsampling the batch size for rare words
- Removing stop words from the vocabulary entirely
- Splitting the corpus into sub-corpora by word frequency
Correct answer: Randomly discarding high-frequency words during training to reduce noise and speed training
Very common words like 'the' carry little specific semantic signal, so discarding them probabilistically improves both speed and embedding quality.
Question 4: How does the window size hyperparameter affect the semantic properties of learned word embeddings?
- Larger windows capture broader topical similarity; smaller windows capture tighter syntactic similarity (Correct answer)
- Larger windows reduce the vocabulary size
- Smaller windows increase the embedding dimensionality
- Window size has no effect on semantic properties
Correct answer: Larger windows capture broader topical similarity; smaller windows capture tighter syntactic similarity
A small context window (e.g., 2) makes the model focus on immediately adjacent syntactic patterns, while a large window (e.g., 10) captures document-level topic associations.
Question 5: In the context of word embeddings, what is 'retrofitting'?
- Post-hoc adjustment of pre-trained embeddings using semantic lexicons like WordNet (Correct answer)
- Re-training embeddings on a new domain corpus from scratch
- Reducing embedding dimensionality with PCA after training
- Adding positional encodings to static word vectors
Correct answer: Post-hoc adjustment of pre-trained embeddings using semantic lexicons like WordNet
Retrofitting pulls embeddings of synonymous words closer together using graph-based constraints from a semantic lexicon, improving lexical similarity.
Question 6: Which metric is most commonly used to measure the distance between two word embedding vectors?
- Cosine similarity (Correct answer)
- Euclidean distance squared
- Manhattan distance
- Hamming distance
Correct answer: Cosine similarity
Cosine similarity measures the angle between vectors and is preferred because it is invariant to vector magnitude, capturing directional semantic similarity.
Question 7: What does it mean for word embeddings to capture 'syntactic' regularities?
- Vector offsets encode grammatical relationships such as singular-plural or verb tense (Correct answer)
- Embeddings are arranged alphabetically in the vector space
- Syntax trees are stored as adjacency lists within the vectors
- Punctuation tokens receive zero vectors
Correct answer: Vector offsets encode grammatical relationships such as singular-plural or verb tense
Word2vec and GloVe embeddings encode syntactic patterns like run→ran and dog→dogs as consistent directional offsets in the embedding space.
NLP Word Embeddings 3
Which phenomenon explains why 'bank' has different meanings in 'river bank' versus 'savings bank,' and how do static embeddings handle it?