NLP Word Embeddings 4 — Questions and Answers
Question 1: How do contextualized embeddings from ELMo differ fundamentally from word2vec embeddings?
- ELMo produces a different vector for the same word depending on its sentence context using a biLSTM (Correct answer)
- ELMo uses larger fixed vectors trained on a bigger corpus
- ELMo replaces the embedding lookup with a character CNN only
- ELMo applies dot-product attention instead of a context window
Correct answer: ELMo produces a different vector for the same word depending on its sentence context using a biLSTM
ELMo generates dynamic, context-sensitive embeddings by passing the full sentence through a bidirectional LSTM, so 'bank' gets different vectors in different sentences.
Question 2: What is 'transfer learning' in the context of pre-trained word embeddings?
- Using embeddings learned on a large general corpus as the starting point for a domain-specific task (Correct answer)
- Copying weights from one neural layer to another within the same model
- Transferring data between different file formats for embedding storage
- Applying dimensionality reduction before fine-tuning
Correct answer: Using embeddings learned on a large general corpus as the starting point for a domain-specific task
Pre-trained embeddings encode general language knowledge that can be transferred to downstream tasks, especially when task-specific data is scarce.
Question 3: In a bag-of-words document representation, what information is lost compared to using averaged word embeddings?
- Word order and compositionality are discarded in both, but BoW also ignores semantic similarity between different words (Correct answer)
- BoW retains word order while embeddings do not
- BoW captures semantic relationships between synonyms
- Averaged embeddings discard frequency information while BoW preserves it fully
Correct answer: Word order and compositionality are discarded in both, but BoW also ignores semantic similarity between different words
BoW treats vocabulary as a sparse orthogonal space where different words have zero similarity, whereas averaged embeddings encode semantic relatedness between words.
Question 4: What is 'domain adaptation' when applied to word embeddings?
- Further training or fine-tuning general embeddings on in-domain text to capture domain-specific vocabulary and meaning (Correct answer)
- Changing the embedding dimension to match a target model architecture
- Translating embeddings from one language to another using a bilingual dictionary
- Normalizing embedding magnitudes to the unit sphere
Correct answer: Further training or fine-tuning general embeddings on in-domain text to capture domain-specific vocabulary and meaning
General-purpose embeddings may not capture specialized terminology well, so continuing training on domain text (e.g., biomedical papers) adapts them to the target domain.
Question 5: Which technique allows word embeddings from two different languages to be mapped into a shared cross-lingual vector space?
- Learning a linear transformation (rotation) using bilingual anchor word pairs (Correct answer)
- Concatenating the two monolingual embedding matrices
- Re-training both language models jointly on a parallel corpus from scratch
- Applying PCA to both embedding sets independently
Correct answer: Learning a linear transformation (rotation) using bilingual anchor word pairs
Cross-lingual alignment methods like VecMap find a rotation matrix that maps monolingual embeddings into a shared space using bilingual seed lexicons.
Question 6: What does the term 'out-of-vocabulary (OOV)' mean for a word embedding model, and how does fastText mitigate it?
- OOV words were not seen during training; fastText mitigates this by composing embeddings from character n-grams (Correct answer)
- OOV means the word appears too frequently and is downsampled to zero
- fastText uses a larger vocabulary with 10× more entries than word2vec
- fastText queries a dictionary API for unseen words at inference time
Correct answer: OOV words were not seen during training; fastText mitigates this by composing embeddings from character n-grams
FastText can approximate embeddings for OOV words by summing the n-gram vectors of their character substrings, which often share morphemes with known words.
Question 7: What bias problem has been documented in word embeddings trained on large web corpora?
- Embeddings reflect societal biases, e.g., associating 'programmer' more closely with male terms than female terms (Correct answer)
- Embeddings assign zero vectors to minority-language words
- Embeddings converge to identical vectors for all occupational nouns
- Embeddings produce negative cosine similarities for all gendered words
Correct answer: Embeddings reflect societal biases, e.g., associating 'programmer' more closely with male terms than female terms
Bolukbasi et al. (2016) showed that word2vec embeddings encode gender stereotypes present in training corpora, such as gender-occupation associations.
NLP Word Embeddings 4
How do contextualized embeddings from ELMo differ fundamentally from word2vec embeddings?