Natural Language Processing Language Models and Neural NLP

Question 1

What is a word embedding in NLP?

Accepted Answer

A dense, low-dimensional vector representation of a word that captures semantic meaning

Answer

A binary one-hot encoding of words

Answer

A list of synonyms for a word

Answer

A frequency count of a word in a corpus

Question 2

What training objective does Word2Vec's Skip-gram model use?

Accepted Answer

Predict surrounding context words given a center word

Answer

Predict the center word given surrounding context words

Answer

Reconstruct the input sequence from a compressed representation

Answer

Classify a sentence as positive or negative

Question 3

What does the attention mechanism in transformers compute?

Accepted Answer

A weighted sum of value vectors based on the relevance of each key to the query

Answer

The probability distribution over the next token in a sequence

Answer

The gradient of the loss with respect to each weight

Answer

A bag-of-words representation of the input

Question 4

What is the primary innovation of the Transformer architecture over RNNs?

Accepted Answer

It uses self-attention to process all tokens in parallel rather than sequentially

Answer

It uses larger embedding dimensions

Answer

It applies convolutions to text instead of recurrence

Answer

It trains only on labeled data

Question 5

What does BERT's masked language modeling (MLM) pre-training objective do?

Accepted Answer

Randomly masks tokens in the input and trains the model to predict the masked words

Answer

Predicts the next sentence given the current sentence

Answer

Generates text from left to right like a language model

Answer

Classifies whether two sentences are paraphrases

Question 6

What distinguishes GPT-style models from BERT-style models?

Accepted Answer

GPT uses a left-to-right causal (decoder-only) language model, while BERT uses a bidirectional encoder

Answer

GPT is smaller than BERT

Answer

BERT generates text while GPT classifies it

Answer

GPT does not use the transformer architecture