Neural Network Backpropagation and Training 2

Question 1

What is the Adam optimizer and what does it combine?

Accepted Answer

Adaptive learning rates using first and second moment estimates of gradients

Answer

A second-order optimizer using Hessians

Answer

A momentum-only optimizer with decay

Answer

A stochastic optimizer with random restarts

Question 2

What is learning rate warmup in neural network training?

Accepted Answer

Gradually increasing the learning rate at the start of training

Answer

Decreasing the learning rate at the end of training

Answer

Using different learning rates for different layers

Answer

Applying cyclic learning rate schedules

Question 3

What is the purpose of weight initialization in neural networks?

Accepted Answer

To set initial weights appropriately to enable stable gradient flow from the start

Answer

To set weights to zero for symmetry breaking

Answer

To ensure all neurons produce the same initial output

Answer

To minimize the initial loss value

Question 4

What does the loss landscape of a neural network refer to?

Accepted Answer

The high-dimensional surface of the loss as a function of all weights

Answer

The visualization of training/validation loss curves

Answer

The plot of activations across layers

Answer

The distribution of gradient magnitudes

Question 5

What is early stopping as a regularization technique?

Accepted Answer

Halting training when validation loss stops improving to prevent overfitting

Answer

Stopping training when gradients fall below a threshold

Answer

Reducing the learning rate after a fixed number of epochs

Answer

Removing neurons with small weights from the network

Question 6

In which scenario would you use gradient accumulation during training?

Accepted Answer

When GPU memory is too small to fit large effective batch sizes

Answer

When you want to use a very high learning rate

Answer

When the dataset has highly imbalanced classes

Answer

When training with multiple output heads