Deep Learning Transformers and Attention

Question 1

What is the key innovation of the Transformer architecture over RNNs?

Accepted Answer

Self-attention allowing all positions to attend to each other in parallel, with no sequential dependency

Answer

Transformers replace sequential recurrence with self-attention, enabling parallel processing of all sequence positions and capturing long-range dependencies more efficiently.

Question 2

In scaled dot-product attention, what are the queries, keys, and values?

Accepted Answer

Learned linear projections of input representations used to compute attention weights and weighted value sums

Answer

Queries, keys, and values are learned linear projections of input vectors; attention weights are computed as softmax(QK^T/√d_k) and applied to V.

Question 3

What problem do positional encodings solve in Transformer models?

Accepted Answer

Providing token order information since self-attention is permutation-invariant

Answer

Since self-attention treats all positions equally, positional encodings inject sequential order information by adding position-dependent vectors to token embeddings.

Question 4

What is multi-head attention?

Accepted Answer

Running attention in parallel with different learned projections and concatenating results to capture diverse relationships

Answer

Multi-head attention projects queries, keys, and values into multiple lower-dimensional subspaces, computes attention in each, and concatenates the results, capturing different relational patterns.

Question 5

What is the role of the feed-forward sub-layer in each Transformer block?

Accepted Answer

Applying a position-wise non-linear transformation independently to each token's representation

Answer

The feed-forward sub-layer applies the same two-layer MLP to each position independently, adding non-linear transformation capacity beyond what attention alone provides.

Deep Learning Practice Test

Deep Learning Practice Test

Deep Learning Transformers and Attention