Deep Learning Transformers and Attention 2 — Questions and Answers
Question 1: What is the computational complexity of self-attention with respect to sequence length n?
- O(n)
- O(n log n)
- O(n²) (Correct answer)
- O(n³)
Correct answer: O(n²)
Self-attention computes pairwise interactions between all n tokens, requiring O(n²) time and memory, which becomes prohibitive for very long sequences.
Question 2: What distinguishes the decoder from the encoder in a standard Transformer?
- The decoder uses CNN layers; the encoder uses attention
- The decoder adds masked self-attention and cross-attention to encoder outputs, preventing attending to future tokens (Correct answer)
- The decoder has no feed-forward sub-layers
- The decoder applies bidirectional attention over the full target sequence
Correct answer: The decoder adds masked self-attention and cross-attention to encoder outputs, preventing attending to future tokens
The decoder uses masked self-attention (blocking future positions) and cross-attention (over encoder outputs), enabling autoregressive generation while conditioning on source context.
Question 3: In the context of large language models, what is autoregressive generation?
- Generating all output tokens simultaneously
- Producing one token at a time, conditioning each on all previously generated tokens (Correct answer)
- Applying regression to predict continuous output values
- Using a regression loss instead of cross-entropy
Correct answer: Producing one token at a time, conditioning each on all previously generated tokens
Autoregressive generation predicts each token sequentially, feeding previous predictions back as input, enabling open-ended text generation one token at a time.
Question 4: What is layer normalization and why is it preferred in Transformers over batch normalization?
- Layer norm normalizes over the batch dimension; it is preferred for its simplicity
- Layer norm normalizes over feature dimensions within each example, avoiding dependence on batch size and working better for variable-length sequences (Correct answer)
- Layer norm adds learnable scaling only to the first layer
- Layer norm is applied only during inference, not training
Correct answer: Layer norm normalizes over feature dimensions within each example, avoiding dependence on batch size and working better for variable-length sequences
Layer normalization normalizes across the feature dimension of each individual example, making it independent of batch size and well-suited for sequence models with variable lengths.
Question 5: What is the difference between GPT and BERT in terms of architecture and pretraining?
- GPT is an encoder; BERT is a decoder
- GPT uses a decoder-only Transformer pretrained with causal language modeling; BERT uses encoder-only with masked language modeling (Correct answer)
- GPT uses bidirectional attention; BERT uses unidirectional attention
- They use identical architectures with different training datasets
Correct answer: GPT uses a decoder-only Transformer pretrained with causal language modeling; BERT uses encoder-only with masked language modeling
GPT is a decoder-only model trained to predict the next token (causal LM), while BERT is an encoder-only model trained with masked tokens, making them suited for generation vs. understanding tasks respectively.
Question 6: What is cross-attention in a Transformer decoder?
- Attention between adjacent layers in the encoder
- Attention where queries come from the decoder and keys/values come from the encoder, allowing decoder to attend to input context (Correct answer)
- Attention applied across multiple training examples in a batch
- Shared attention weights between encoder and decoder
Correct answer: Attention where queries come from the decoder and keys/values come from the encoder, allowing decoder to attend to input context
Cross-attention lets each decoder position query the encoder's output representations, allowing the decoder to selectively focus on relevant parts of the input sequence.
Deep Learning Transformers and Attention 2
What is the computational complexity of self-attention with respect to sequence length n?