Neural Network Activation Functions and Optimization 3

Question 1

What is a saddle point in the loss landscape of a neural network?

Accepted Answer

A critical point that is a minimum in some directions and a maximum in others

Answer

A saddle point is a critical point (zero gradient) that is a local minimum in some directions and a local maximum in others; neural network optimization landscapes contain many saddle points.

Question 2

What is cosine annealing as a learning rate schedule?

Accepted Answer

Annealing the learning rate following a cosine function from an initial value toward a minimum

Answer

Cosine annealing decays the learning rate following a half-cosine curve from η_max to η_min, often with warm restarts, which helps the optimizer escape sharp minima and find flatter ones.

Question 3

What does SELU (Scaled ELU) achieve that makes it 'self-normalizing'?

Accepted Answer

With proper initialization, it drives activations to zero mean and unit variance automatically across layers

Answer

SELU uses carefully chosen scale and alpha parameters that, with Lecun normal initialization, maintain zero mean and unit variance of activations throughout the network without explicit normalization layers.

Question 4

What is the difference between weight decay and L2 regularization in modern optimizers like Adam?

Accepted Answer

In adaptive optimizers, L2 regularization interacts with the adaptive learning rate scaling while weight decay applies directly to weights

Answer

In adaptive optimizers like Adam, L2 regularization adds the penalty to the gradient before adaptive scaling, inadvertently decoupling it from the scaling; AdamW applies weight decay directly to weights, giving proper regularization.

Question 5

What is Nesterov accelerated gradient (NAG) and how does it improve standard momentum?

Accepted Answer

It computes the gradient at the anticipated next position rather than the current position, giving a look-ahead update

Answer

NAG evaluates the gradient at the point the momentum would take the parameters to (look-ahead), rather than the current point, providing a corrective update that reduces oscillations and improves convergence.

Neural Network Practice Test

Neural Network Practice Test

Neural Network Activation Functions and Optimization 3