Deep Learning Transformers and Attention 1 — Questions and Answers
Question 1: What is the key innovation of the Transformer architecture over RNNs?
- Use of convolutional operations for sequence modeling
- Self-attention allowing all positions to attend to each other in parallel, with no sequential dependency (Correct answer)
- Replacing activation functions with gating mechanisms
- Using separate encoders for each input token
Correct answer: Self-attention allowing all positions to attend to each other in parallel, with no sequential dependency
Transformers replace sequential recurrence with self-attention, enabling parallel processing of all sequence positions and capturing long-range dependencies more efficiently.
Question 2: In scaled dot-product attention, what are the queries, keys, and values?
- Database lookup operations unrelated to neural networks
- Learned linear projections of input representations used to compute attention weights and weighted value sums (Correct answer)
- The input, hidden, and output states of an RNN
- Three separate embedding matrices for different word senses
Correct answer: Learned linear projections of input representations used to compute attention weights and weighted value sums
Queries, keys, and values are learned linear projections of input vectors; attention weights are computed as softmax(QK^T/√d_k) and applied to V.
Question 3: What problem do positional encodings solve in Transformer models?
- Preventing attention weights from becoming uniform
- Providing token order information since self-attention is permutation-invariant (Correct answer)
- Reducing the computational cost of multi-head attention
- Normalizing embedding magnitudes across the vocabulary
Correct answer: Providing token order information since self-attention is permutation-invariant
Since self-attention treats all positions equally, positional encodings inject sequential order information by adding position-dependent vectors to token embeddings.
Question 4: What is multi-head attention?
- Applying attention multiple times sequentially
- Running attention in parallel with different learned projections and concatenating results to capture diverse relationships (Correct answer)
- Using a different attention function for each decoder layer
- Averaging attention patterns from multiple training runs
Correct answer: Running attention in parallel with different learned projections and concatenating results to capture diverse relationships
Multi-head attention projects queries, keys, and values into multiple lower-dimensional subspaces, computes attention in each, and concatenates the results, capturing different relational patterns.
Question 5: What is the role of the feed-forward sub-layer in each Transformer block?
- Computing attention weights between tokens
- Applying a position-wise non-linear transformation independently to each token's representation (Correct answer)
- Normalizing attention outputs across the sequence
- Encoding positional information into token embeddings
Correct answer: Applying a position-wise non-linear transformation independently to each token's representation
The feed-forward sub-layer applies the same two-layer MLP to each position independently, adding non-linear transformation capacity beyond what attention alone provides.
Question 6: What does BERT stand for and what training objective made it influential?
- Bidirectional Encoder Recurrent Transformer; next-word prediction
- Bidirectional Encoder Representations from Transformers; masked language modeling and next-sentence prediction (Correct answer)
- Broadly Encoded Representation Transformer; contrastive learning
- Batch-Efficient Recurrent Transformer; sequence classification
Correct answer: Bidirectional Encoder Representations from Transformers; masked language modeling and next-sentence prediction
BERT pretrains deep bidirectional representations using masked language modeling (predicting masked tokens) and next-sentence prediction, producing rich contextual embeddings.
Deep Learning Transformers and Attention 1
What is the key innovation of the Transformer architecture over RNNs?