Deep Learning Transformers and Attention

Question 1

What is the key innovation of the Transformer architecture over RNNs?

Accepted Answer

Self-attention allowing all positions to attend to each other in parallel, with no sequential dependency

Answer

Use of convolutional operations for sequence modeling

Answer

Replacing activation functions with gating mechanisms

Answer

Using separate encoders for each input token

Question 2

In scaled dot-product attention, what are the queries, keys, and values?

Accepted Answer

Learned linear projections of input representations used to compute attention weights and weighted value sums

Answer

Database lookup operations unrelated to neural networks

Answer

The input, hidden, and output states of an RNN

Answer

Three separate embedding matrices for different word senses

Question 3

What problem do positional encodings solve in Transformer models?

Accepted Answer

Providing token order information since self-attention is permutation-invariant

Answer

Preventing attention weights from becoming uniform

Answer

Reducing the computational cost of multi-head attention

Answer

Normalizing embedding magnitudes across the vocabulary

Question 4

What is multi-head attention?

Accepted Answer

Running attention in parallel with different learned projections and concatenating results to capture diverse relationships

Answer

Applying attention multiple times sequentially

Answer

Using a different attention function for each decoder layer

Answer

Averaging attention patterns from multiple training runs

Question 5

What is the role of the feed-forward sub-layer in each Transformer block?

Accepted Answer

Applying a position-wise non-linear transformation independently to each token's representation

Answer

Computing attention weights between tokens

Answer

Normalizing attention outputs across the sequence

Answer

Encoding positional information into token embeddings

Question 6

What does BERT stand for and what training objective made it influential?

Accepted Answer

Bidirectional Encoder Representations from Transformers; masked language modeling and next-sentence prediction

Answer

Bidirectional Encoder Recurrent Transformer; next-word prediction

Answer

Broadly Encoded Representation Transformer; contrastive learning

Answer

Batch-Efficient Recurrent Transformer; sequence classification