Deep Learning (Data Scientist) 3 — Questions and Answers
Question 1: A data scientist training an RNN on long text sequences finds gradients shrinking toward zero in early timesteps. Which architecture change best addresses this?
- Replace vanilla RNN cells with LSTM or GRU cells (Correct answer)
- Increase the batch size
- Use mean squared error instead of cross-entropy
- Remove the embedding layer
Correct answer: Replace vanilla RNN cells with LSTM or GRU cells
LSTM and GRU gating mechanisms preserve gradient flow across long sequences, mitigating the vanishing gradient problem.
Question 2: In the Transformer architecture, what is the primary purpose of the self-attention mechanism?
- To weigh the relevance of every token to every other token in a sequence (Correct answer)
- To reduce the number of parameters compared to convolutions
- To normalize the input embeddings
- To prevent overfitting by dropping random tokens
Correct answer: To weigh the relevance of every token to every other token in a sequence
Self-attention computes pairwise relevance scores so each token's representation incorporates context from the entire sequence.
Question 3: Why do Transformers require positional encodings?
- Self-attention is permutation-invariant and has no inherent sense of token order (Correct answer)
- They reduce the memory footprint of attention
- They replace the need for a softmax layer
- They speed up backpropagation through time
Correct answer: Self-attention is permutation-invariant and has no inherent sense of token order
Without positional encodings, attention treats the input as an unordered set, losing sequence order information.
Question 4: A convolutional layer has a 3x3 kernel, stride 1, and 'same' padding applied to a 64x64 input. What is the spatial size of the output?
- 64x64 (Correct answer)
- 62x62
- 32x32
- 66x66
Correct answer: 64x64
'Same' padding with stride 1 preserves the input's spatial dimensions.
Question 5: Which optimizer combines momentum with per-parameter adaptive learning rates derived from first and second moment estimates?
- Adam (Correct answer)
- Stochastic gradient descent
- Newton's method
- Coordinate descent
Correct answer: Adam
Adam maintains exponential moving averages of both gradients and squared gradients to adapt each parameter's step size.
Question 6: When training a deep network, exploding gradients cause the loss to become NaN. Which is the most direct remedy?
- Apply gradient clipping (Correct answer)
- Increase the number of epochs
- Switch from ReLU to sigmoid activations everywhere
- Disable weight initialization
Correct answer: Apply gradient clipping
Gradient clipping caps the gradient norm, preventing runaway updates that produce numerical overflow.
Question 7: In a variational autoencoder (VAE), what role does the KL-divergence term in the loss play?
- It regularizes the latent distribution toward a chosen prior, typically a standard normal (Correct answer)
- It measures pixel-wise reconstruction error
- It penalizes large weights in the encoder
- It enforces sparsity in the decoder outputs
Correct answer: It regularizes the latent distribution toward a chosen prior, typically a standard normal
The KL term pushes the learned latent distribution toward the prior, making the latent space smooth and sampleable.
Deep Learning (Data Scientist) 3
A data scientist training an RNN on long text sequences finds gradients shrinking toward zero in early timesteps.
Which architecture change best addresses this?