Deep Learning Transformers and Attention 2

Question 1

What is the computational complexity of self-attention with respect to sequence length n?

Accepted Answer

O(n²)

Answer

Self-attention computes pairwise interactions between all n tokens, requiring O(n²) time and memory, which becomes prohibitive for very long sequences.

Question 2

What distinguishes the decoder from the encoder in a standard Transformer?

Accepted Answer

The decoder adds masked self-attention and cross-attention to encoder outputs, preventing attending to future tokens

Answer

The decoder uses masked self-attention (blocking future positions) and cross-attention (over encoder outputs), enabling autoregressive generation while conditioning on source context.

Question 3

In the context of large language models, what is autoregressive generation?

Accepted Answer

Producing one token at a time, conditioning each on all previously generated tokens

Answer

Autoregressive generation predicts each token sequentially, feeding previous predictions back as input, enabling open-ended text generation one token at a time.

Question 4

What is layer normalization and why is it preferred in Transformers over batch normalization?

Accepted Answer

Layer norm normalizes over feature dimensions within each example, avoiding dependence on batch size and working better for variable-length sequences

Answer

Layer normalization normalizes across the feature dimension of each individual example, making it independent of batch size and well-suited for sequence models with variable lengths.

Question 5

What is the difference between GPT and BERT in terms of architecture and pretraining?

Accepted Answer

GPT uses a decoder-only Transformer pretrained with causal language modeling; BERT uses encoder-only with masked language modeling

Answer

GPT is a decoder-only model trained to predict the next token (causal LM), while BERT is an encoder-only model trained with masked tokens, making them suited for generation vs. understanding tasks respectively.

Deep Learning Practice Test

Deep Learning Practice Test

Deep Learning Transformers and Attention 2