NLP Word Embeddings 5

Question 1

What does 'debiasing' a word embedding mean in practice?

Accepted Answer

Projecting embeddings to remove a defined bias direction, e.g., a gender axis, while preserving other semantic properties

Answer

Bolukbasi et al. proposed identifying the gender subspace via PCA and projecting neutral words onto the orthogonal complement to neutralize gender bias.

Question 2

How are word embeddings typically initialized before fine-tuning on a downstream task?

Accepted Answer

Loaded from a pre-trained model (e.g., GloVe or word2vec) rather than random initialization

Answer

Using pre-trained embeddings as initialization leverages general language knowledge and speeds convergence, especially on small datasets.

Question 3

Why is Principal Component Analysis (PCA) sometimes applied to word embeddings after training?

Accepted Answer

To reduce embedding dimensionality, remove noise, and sometimes improve downstream task performance

Answer

PCA can compress 300-dimensional embeddings into fewer dimensions while preserving most variance, reducing memory and sometimes denoising the representation.

Question 4

What is the 'hubness problem' in high-dimensional word embedding spaces?

Accepted Answer

A small number of points become nearest neighbors of many other points, skewing nearest-neighbor retrieval

Answer

In high-dimensional spaces, some vectors systematically appear in many k-NN lists ('hubs'), degrading retrieval quality in cross-lingual or similarity search tasks.

Question 5

In the word2vec hierarchical softmax approach, what data structure is used to speed up vocabulary-scale predictions?

Accepted Answer

A Huffman binary tree where frequent words are placed at shallower nodes

Answer

Hierarchical softmax encodes the vocabulary in a Huffman tree; each word's probability is computed as a product of binary decisions along the path to its leaf, reducing cost from O(V) to O(log V).

NLP Practice Test

NLP Practice Test

NLP Word Embeddings 5