Machine Learning Neural Networks 5

Question 1

What is the attention mechanism in transformer-based neural networks?

Accepted Answer

A method to weight the relevance of different input positions when producing each output

Answer

Attention computes a weighted sum of values based on query-key similarities, allowing the model to focus on the most relevant input positions.

Question 2

What distinguishes a recurrent neural network (RNN) from a feedforward network?

Accepted Answer

RNNs have connections that feed hidden state back into the next time step

Answer

RNNs maintain a hidden state passed between time steps, enabling them to model sequential dependencies in time-series or text data.

Question 3

Which regularization technique adds the sum of absolute values of weights to the loss function?

Accepted Answer

L1 regularization

Answer

L1 regularization (Lasso) penalizes the sum of absolute weight values, encouraging sparsity by driving some weights exactly to zero.

Question 4

What does the term 'dead neuron' refer to when using ReLU activations?

Accepted Answer

A neuron that always outputs zero because its weights push its input permanently negative

Answer

A dead ReLU neuron receives only negative inputs, always outputting zero with zero gradient, meaning it can never recover during training.

Question 5

In neural architecture search (NAS), what is the goal?

Accepted Answer

To automatically discover optimal neural network architectures using search algorithms

Answer

NAS automates the design of neural network architectures by searching over possible configurations using techniques like reinforcement learning or evolutionary algorithms.

Machine Learning Practice Test

Machine Learning Practice Test

Machine Learning Neural Networks 5