Machine Learning Neural Networks 5 — Questions and Answers
Question 1: What is the attention mechanism in transformer-based neural networks?
- A regularization technique that focuses training on hard examples
- A method to weight the relevance of different input positions when producing each output (Correct answer)
- A pruning strategy that removes low-activation neurons
- A gating mechanism borrowed directly from LSTM cells
Correct answer: A method to weight the relevance of different input positions when producing each output
Attention computes a weighted sum of values based on query-key similarities, allowing the model to focus on the most relevant input positions.
Question 2: What distinguishes a recurrent neural network (RNN) from a feedforward network?
- RNNs use convolutional filters while feedforward networks do not
- RNNs have connections that feed hidden state back into the next time step (Correct answer)
- RNNs have no hidden layers and process data in one pass
- RNNs use different activation functions than feedforward networks
Correct answer: RNNs have connections that feed hidden state back into the next time step
RNNs maintain a hidden state passed between time steps, enabling them to model sequential dependencies in time-series or text data.
Question 3: Which regularization technique adds the sum of absolute values of weights to the loss function?
- L2 regularization
- Dropout
- L1 regularization (Correct answer)
- Weight normalization
Correct answer: L1 regularization
L1 regularization (Lasso) penalizes the sum of absolute weight values, encouraging sparsity by driving some weights exactly to zero.
Question 4: What does the term 'dead neuron' refer to when using ReLU activations?
- A neuron whose weights have not been initialized
- A neuron that always outputs zero because its weights push its input permanently negative (Correct answer)
- A neuron that outputs the same value regardless of input
- A neuron removed by the dropout process
Correct answer: A neuron that always outputs zero because its weights push its input permanently negative
A dead ReLU neuron receives only negative inputs, always outputting zero with zero gradient, meaning it can never recover during training.
Question 5: In neural architecture search (NAS), what is the goal?
- To manually design the best network topology for a task
- To automatically discover optimal neural network architectures using search algorithms (Correct answer)
- To prune a pretrained model to the smallest size
- To search for the best learning rate schedule
Correct answer: To automatically discover optimal neural network architectures using search algorithms
NAS automates the design of neural network architectures by searching over possible configurations using techniques like reinforcement learning or evolutionary algorithms.
Question 6: What is the primary advantage of using leaky ReLU over standard ReLU?
- It is computationally cheaper to compute
- It allows a small gradient for negative inputs, preventing dead neurons (Correct answer)
- It bounds outputs between 0 and 1 like sigmoid
- It is differentiable everywhere including at zero
Correct answer: It allows a small gradient for negative inputs, preventing dead neurons
Leaky ReLU passes a small fraction (e.g., 0.01) of negative input values, ensuring non-zero gradients for negative activations and mitigating the dead neuron problem.
Question 7: What does 'data augmentation' achieve during neural network training?
- It increases the number of model parameters
- It artificially expands training data variety to improve generalization (Correct answer)
- It removes noise from training labels
- It speeds up convergence by normalizing the dataset
Correct answer: It artificially expands training data variety to improve generalization
Data augmentation applies transformations (flipping, cropping, rotation) to training examples, increasing effective dataset size and reducing overfitting.
Machine Learning Neural Networks 5
What is the attention mechanism in transformer-based neural networks?