Deep Learning Transformers and Attention 2

Question 1

What is the computational complexity of self-attention with respect to sequence length n?

Accepted Answer

O(n²)

Answer

O(n)

Answer

O(n log n)

Answer

O(n³)

Question 2

What distinguishes the decoder from the encoder in a standard Transformer?

Accepted Answer

The decoder adds masked self-attention and cross-attention to encoder outputs, preventing attending to future tokens

Answer

The decoder uses CNN layers; the encoder uses attention

Answer

The decoder has no feed-forward sub-layers

Answer

The decoder applies bidirectional attention over the full target sequence

Question 3

In the context of large language models, what is autoregressive generation?

Accepted Answer

Producing one token at a time, conditioning each on all previously generated tokens

Answer

Generating all output tokens simultaneously

Answer

Applying regression to predict continuous output values

Answer

Using a regression loss instead of cross-entropy

Question 4

What is layer normalization and why is it preferred in Transformers over batch normalization?

Accepted Answer

Layer norm normalizes over feature dimensions within each example, avoiding dependence on batch size and working better for variable-length sequences

Answer

Layer norm normalizes over the batch dimension; it is preferred for its simplicity

Answer

Layer norm adds learnable scaling only to the first layer

Answer

Layer norm is applied only during inference, not training

Question 5

What is the difference between GPT and BERT in terms of architecture and pretraining?

Accepted Answer

GPT uses a decoder-only Transformer pretrained with causal language modeling; BERT uses encoder-only with masked language modeling

Answer

GPT is an encoder; BERT is a decoder

Answer

GPT uses bidirectional attention; BERT uses unidirectional attention

Answer

They use identical architectures with different training datasets

Question 6

What is cross-attention in a Transformer decoder?

Accepted Answer

Attention where queries come from the decoder and keys/values come from the encoder, allowing decoder to attend to input context

Answer

Attention between adjacent layers in the encoder

Answer

Attention applied across multiple training examples in a batch

Answer

Shared attention weights between encoder and decoder