Training and Optimization Flashcards

What does the learning rate scheduler 'ReduceLROnPlateau' do when validation loss stops improving?

Answer: Reduces the learning rate by a factor

ReduceLROnPlateau monitors a metric and reduces the learning rate by a specified factor when no improvement is seen for a patience number of epochs.

Which technique adds Gaussian noise to the gradients during CNN training to help escape sharp local minima?

Answer: Stochastic gradient noise

Stochastic gradient noise injects Gaussian noise into gradients during optimization, helping the model escape sharp local minima and find flatter, more generalizable optima.

In CNN training, what does 'warm restarts' refer to in the context of learning rate schedules?

Answer: Periodically resetting the learning rate to a high value then annealing it

Warm restarts (SGDR) periodically resets the learning rate to a maximum value and then follows a cosine annealing schedule, helping the model explore different loss landscape regions.

What is the primary benefit of using mixed-precision training (FP16 + FP32) in CNN optimization?

Answer: Reduced training time and memory usage with minimal accuracy loss

Mixed-precision training uses 16-bit floats for most operations and 32-bit for critical accumulations, significantly reducing memory footprint and speeding up computation on modern GPUs.

During CNN training, which regularization method randomly drops entire feature maps rather than individual neurons?

Answer: DropBlock

DropBlock drops contiguous regions of feature maps, which is more effective for CNNs than standard dropout because adjacent units in feature maps tend to be correlated.

What problem does gradient checkpointing solve in CNN training?

Answer: High memory usage during backpropagation by recomputing activations

Gradient checkpointing reduces memory by not storing all intermediate activations; instead, it recomputes them during the backward pass, trading compute time for memory savings.

Which optimizer introduces a per-parameter adaptive learning rate based on the second moment of gradients?

Answer: Adam

Adam maintains both first moment (mean) and second moment (uncentered variance) estimates of gradients, providing adaptive per-parameter learning rates with bias correction.