Natural Language Processing Language Models and Neural NLP

Question 1

What is a word embedding in NLP?

Accepted Answer

A dense, low-dimensional vector representation of a word that captures semantic meaning

Answer

Word embeddings map words to dense real-valued vectors in a continuous space where semantically similar words are geometrically close.

Question 2

What training objective does Word2Vec's Skip-gram model use?

Accepted Answer

Predict surrounding context words given a center word

Answer

Skip-gram trains a model to predict the words surrounding a given target word, learning embeddings that encode co-occurrence patterns.

Question 3

What does the attention mechanism in transformers compute?

Accepted Answer

A weighted sum of value vectors based on the relevance of each key to the query

Answer

Attention computes compatibility scores between a query and all keys, uses softmax to get weights, and produces a weighted sum of corresponding values.

Question 4

What is the primary innovation of the Transformer architecture over RNNs?

Accepted Answer

It uses self-attention to process all tokens in parallel rather than sequentially

Answer

Transformers process the entire sequence simultaneously via self-attention, enabling parallelization and capturing long-range dependencies without vanishing gradients.

Question 5

What does BERT's masked language modeling (MLM) pre-training objective do?

Accepted Answer

Randomly masks tokens in the input and trains the model to predict the masked words

Answer

MLM masks 15% of input tokens and trains BERT to reconstruct them, enabling bidirectional context learning.

Natural Language Processing Practice Test

Natural Language Processing Practice Test

Natural Language Processing Language Models and Neural NLP