Deep Learning and Neural Networks Flashcards

What is the role of the attention mechanism in transformer-based deep learning models?

Answer: Allow the model to weigh the relevance of each input token when producing an output

Attention computes a weighted sum of value vectors, where weights reflect how relevant each key-query pair is, enabling long-range dependency modeling.

Which optimizer adaptively adjusts learning rates for each parameter using estimates of first and second moments of the gradients?

Answer: Adam

Adam combines momentum (first moment) and RMSProp (second moment) to compute adaptive per-parameter learning rates, making it robust across many tasks.

In a generative adversarial network (GAN), what does the discriminator try to do?

Answer: Distinguish real data from generated data

The discriminator is a binary classifier trained to output high probability for real samples and low probability for samples generated by the generator.

What is batch normalization primarily used for in deep neural networks?

Answer: Normalizing layer inputs to stabilize and accelerate training

Batch normalization normalizes each mini-batch's activations to zero mean and unit variance, reducing internal covariate shift and allowing higher learning rates.

Which regularization technique randomly deactivates a fraction of neurons during each training step to reduce overfitting?

Answer: Dropout

Dropout randomly zeros neuron activations with probability p during training, forcing the network to learn redundant representations and reducing co-adaptation.

In deep learning, what does the term 'epoch' refer to?

Answer: One complete pass through the entire training dataset

An epoch is one full cycle through all training samples; multiple epochs allow the model to refine its weights by seeing the data repeatedly.