NLP Word Embeddings 5 — Questions and Answers
Question 1: What does 'debiasing' a word embedding mean in practice?
- Projecting embeddings to remove a defined bias direction, e.g., a gender axis, while preserving other semantic properties (Correct answer)
- Re-training on a balanced corpus with equal gender representation
- Replacing biased words with gender-neutral synonyms in the vocabulary
- Increasing embedding dimensionality to dilute the bias signal
Correct answer: Projecting embeddings to remove a defined bias direction, e.g., a gender axis, while preserving other semantic properties
Bolukbasi et al. proposed identifying the gender subspace via PCA and projecting neutral words onto the orthogonal complement to neutralize gender bias.
Question 2: How are word embeddings typically initialized before fine-tuning on a downstream task?
- Loaded from a pre-trained model (e.g., GloVe or word2vec) rather than random initialization (Correct answer)
- Set to the identity matrix
- Initialized to the one-hot encoding of each vocabulary index
- Sampled from a Laplace distribution centered at one
Correct answer: Loaded from a pre-trained model (e.g., GloVe or word2vec) rather than random initialization
Using pre-trained embeddings as initialization leverages general language knowledge and speeds convergence, especially on small datasets.
Question 3: Why is Principal Component Analysis (PCA) sometimes applied to word embeddings after training?
- To reduce embedding dimensionality, remove noise, and sometimes improve downstream task performance (Correct answer)
- To convert continuous embeddings into binary codes
- To increase embedding dimensionality for more expressive representations
- To convert embeddings from a dense to a sparse format
Correct answer: To reduce embedding dimensionality, remove noise, and sometimes improve downstream task performance
PCA can compress 300-dimensional embeddings into fewer dimensions while preserving most variance, reducing memory and sometimes denoising the representation.
Question 4: What is the 'hubness problem' in high-dimensional word embedding spaces?
- A small number of points become nearest neighbors of many other points, skewing nearest-neighbor retrieval (Correct answer)
- The embedding space becomes too sparse to compute cosine similarity
- Embedding vectors converge to a single hub vector during training
- The vocabulary grows unboundedly as training progresses
Correct answer: A small number of points become nearest neighbors of many other points, skewing nearest-neighbor retrieval
In high-dimensional spaces, some vectors systematically appear in many k-NN lists ('hubs'), degrading retrieval quality in cross-lingual or similarity search tasks.
Question 5: In the word2vec hierarchical softmax approach, what data structure is used to speed up vocabulary-scale predictions?
- A Huffman binary tree where frequent words are placed at shallower nodes (Correct answer)
- A hash table mapping words to fixed-size buckets
- A prefix trie over character sequences
- A sorted array with binary search over embedding norms
Correct answer: A Huffman binary tree where frequent words are placed at shallower nodes
Hierarchical softmax encodes the vocabulary in a Huffman tree; each word's probability is computed as a product of binary decisions along the path to its leaf, reducing cost from O(V) to O(log V).
Question 6: What is a 'sentence embedding,' and how does it differ from averaging individual word embeddings?
- A sentence embedding is a single vector representing the whole sentence's meaning, often capturing word order and composition that simple averaging ignores (Correct answer)
- Sentence embeddings are identical to averaged word embeddings but stored as sparse vectors
- Sentence embeddings concatenate all word vectors into a single long vector
- Sentence embeddings are obtained by taking the maximum word vector component-wise
Correct answer: A sentence embedding is a single vector representing the whole sentence's meaning, often capturing word order and composition that simple averaging ignores
Models like Sentence-BERT produce sentence embeddings via fine-tuned transformers that encode word order and inter-word relationships, unlike naive averaging which is order-invariant.
Question 7: Which of the following best describes 'sparse' versus 'dense' word representations?
- Sparse representations (e.g., one-hot, TF-IDF) have mostly zero values; dense embeddings (e.g., word2vec) are short vectors with all non-zero values encoding distributed meaning (Correct answer)
- Sparse representations are smaller in memory than dense ones for any vocabulary size
- Dense representations require a lookup table while sparse ones are computed on the fly
- Sparse embeddings outperform dense embeddings on all NLP benchmarks
Correct answer: Sparse representations (e.g., one-hot, TF-IDF) have mostly zero values; dense embeddings (e.g., word2vec) are short vectors with all non-zero values encoding distributed meaning
One-hot vectors are as long as the vocabulary (often 100k+) with a single 1, while dense embeddings pack semantic information into compact 50–300 dimensional vectors with no zero structure.
NLP Word Embeddings 5
What does 'debiasing' a word embedding mean in practice?