NLP Word Embeddings 3

Question 1

Which phenomenon explains why 'bank' has different meanings in 'river bank' versus 'savings bank,' and how do static embeddings handle it?

Accepted Answer

Polysemy; static embeddings assign one fixed vector per word regardless of context

Answer

Static embeddings like word2vec average all senses of a polysemous word into a single vector, losing context-specific meaning.

Question 2

What role does the projection layer play in a word2vec neural network?

Accepted Answer

It maps one-hot input vectors to dense lower-dimensional embedding vectors

Answer

The projection layer is a simple linear lookup that converts sparse one-hot word representations into dense embedding vectors without a non-linearity.

Question 3

What is 'subsampling of frequent words' in word2vec, and why is it used?

Accepted Answer

Randomly discarding high-frequency words during training to reduce noise and speed training

Answer

Very common words like 'the' carry little specific semantic signal, so discarding them probabilistically improves both speed and embedding quality.

Question 4

How does the window size hyperparameter affect the semantic properties of learned word embeddings?

Accepted Answer

Larger windows capture broader topical similarity; smaller windows capture tighter syntactic similarity

Answer

A small context window (e.g., 2) makes the model focus on immediately adjacent syntactic patterns, while a large window (e.g., 10) captures document-level topic associations.

Question 5

In the context of word embeddings, what is 'retrofitting'?

Accepted Answer

Post-hoc adjustment of pre-trained embeddings using semantic lexicons like WordNet

Answer

Retrofitting pulls embeddings of synonymous words closer together using graph-based constraints from a semantic lexicon, improving lexical similarity.

NLP Practice Test

NLP Practice Test

NLP Word Embeddings 3