AML Deep Learning & Neural Networks 3 — Questions and Answers
Question 1: What is 'gradient checkpointing' used for in deep learning training?
- Clipping gradients to prevent explosion
- Trading compute for memory by recomputing activations during the backward pass (Correct answer)
- Saving model weights at regular intervals
- Monitoring gradient norms for debugging
Correct answer: Trading compute for memory by recomputing activations during the backward pass
Gradient checkpointing discards intermediate activations and recomputes them during backpropagation, reducing memory at the cost of extra computation.
Question 2: In the LSTM architecture, which gate controls how much of the previous cell state is retained?
- Input gate
- Output gate
- Forget gate (Correct answer)
- Peephole gate
Correct answer: Forget gate
The forget gate applies a sigmoid-activated mask to the previous cell state, determining what proportion of historical information to carry forward.
Question 3: What distinguishes 'instance normalization' from 'batch normalization'?
- Instance normalization normalizes across the batch dimension; batch normalization normalizes per sample
- Instance normalization normalizes each sample's spatial dimensions independently of other samples in the batch (Correct answer)
- Instance normalization uses learnable parameters; batch normalization does not
- Instance normalization is applied after activation functions; batch normalization before
Correct answer: Instance normalization normalizes each sample's spatial dimensions independently of other samples in the batch
Instance normalization computes mean and variance per sample per channel, making it batch-size-independent and preferred in style transfer tasks.
Question 4: Which concept in neural architecture search (NAS) allows gradient-based optimization of the architecture by making the search space continuous?
- Evolutionary search
- DARTS (Differentiable Architecture Search) (Correct answer)
- Reinforcement learning controller
- Random search with early stopping
Correct answer: DARTS (Differentiable Architecture Search)
DARTS relaxes the discrete architecture choice into a continuous softmax over candidate operations, enabling end-to-end gradient optimization.
Question 5: In contrastive learning (e.g., SimCLR), what is the role of the projection head?
- To produce final class predictions during inference
- To map representations to a space where the contrastive loss is applied, then discarded at downstream fine-tuning (Correct answer)
- To align positional encodings across augmented views
- To normalize representations before passing them to the backbone
Correct answer: To map representations to a space where the contrastive loss is applied, then discarded at downstream fine-tuning
The projection head is a small MLP used only during pretraining to compute the contrastive loss; the backbone encoder is used for downstream tasks.
Question 6: What problem does 'dying ReLU' refer to in deep networks?
- Neurons that produce excessively large activations during forward pass
- Neurons whose pre-activations are consistently negative, causing them to always output zero and stop learning (Correct answer)
- Overfitting caused by too many ReLU activations
- ReLU units that produce NaN values due to numerical overflow
Correct answer: Neurons whose pre-activations are consistently negative, causing them to always output zero and stop learning
When a ReLU neuron's input is always negative its gradient is zero, permanently preventing weight updates and effectively killing that neuron.
Question 7: Which technique is used to visualize what spatial patterns a specific CNN filter has learned to detect?
- t-SNE projection of filter weights
- Activation maximization via gradient ascent on the input (Correct answer)
- Principal component analysis of the feature map
- Monte Carlo sampling of input patches
Correct answer: Activation maximization via gradient ascent on the input
Activation maximization synthesizes an input image by ascending the gradient of a target neuron's activation, revealing its preferred pattern.
AML Deep Learning & Neural Networks 3
What is 'gradient checkpointing' used for in deep learning training?