Neural Network Activation Functions and Optimization 3

Question 1

What is a saddle point in the loss landscape of a neural network?

Accepted Answer

A critical point that is a minimum in some directions and a maximum in others

Answer

A local minimum surrounded by steep walls

Answer

The global minimum of the loss function

Answer

A flat region where gradients are zero uniformly

Question 2

What is cosine annealing as a learning rate schedule?

Accepted Answer

Annealing the learning rate following a cosine function from an initial value toward a minimum

Answer

Linearly decreasing the learning rate each epoch

Answer

Using a cosine activation function instead of learning rate decay

Answer

Applying cyclical momentum with cosine periods

Question 3

What does SELU (Scaled ELU) achieve that makes it 'self-normalizing'?

Accepted Answer

With proper initialization, it drives activations to zero mean and unit variance automatically across layers

Answer

It normalizes gradients to unit variance

Answer

It applies batch normalization at each step

Answer

It scales weights to lie within a fixed range

Question 4

What is the difference between weight decay and L2 regularization in modern optimizers like Adam?

Accepted Answer

In adaptive optimizers, L2 regularization interacts with the adaptive learning rate scaling while weight decay applies directly to weights

Answer

They are mathematically identical in all optimizers

Answer

Weight decay only applies to biases; L2 to all parameters

Answer

They differ only in the magnitude of the penalty term

Question 5

What is Nesterov accelerated gradient (NAG) and how does it improve standard momentum?

Accepted Answer

It computes the gradient at the anticipated next position rather than the current position, giving a look-ahead update

Answer

It applies momentum only when gradients are large

Answer

It uses second-order momentum for faster convergence

Answer

It applies different momentum values to each layer

Question 6

What is the purpose of the temperature parameter in the softmax function during knowledge distillation?

Accepted Answer

To soften the probability distribution so the teacher's output provides more information about inter-class similarities

Answer

To control the learning rate during distillation

Answer

To scale the student network's logits before computing loss

Answer

To control the mixing ratio of hard and soft targets