AML Reinforcement Learning & Decision Making 2

Question 1

In Q-learning, what does the Q-value Q(s, a) represent?

Accepted Answer

The expected cumulative discounted reward when taking action a in state s and following the optimal policy thereafter

Answer

Q(s, a) is the expected total discounted return from taking action a in state s and then acting optimally, combining immediate reward with future expected returns.

Question 2

What fundamentally distinguishes SARSA from Q-learning?

Accepted Answer

SARSA is on-policy and updates using the actual next action taken; Q-learning is off-policy and updates using the greedy max-Q action

Answer

SARSA (on-policy) updates Q-values using the action actually taken by the behavior policy, while Q-learning (off-policy) always bootstraps from the greedy maximum Q-value action.

Question 3

What is the primary motivation for using Deep Q-Networks (DQN) over tabular Q-learning?

Accepted Answer

DQN uses neural networks to approximate Q-values, enabling generalization over large or continuous state spaces

Answer

DQN approximates Q-values with neural networks, enabling generalization across similar states and making RL tractable in high-dimensional or continuous state spaces where a Q-table is infeasible.

Question 4

What problem does experience replay solve in Deep Q-Networks?

Accepted Answer

It reduces correlation between consecutive training samples, stabilizing neural network learning

Answer

Experience replay stores transitions in a buffer and randomly samples mini-batches, breaking the temporal correlation between consecutive samples that would otherwise cause unstable, divergent training.

Question 5

In the Actor-Critic architecture, what is the primary role of the Critic?

Accepted Answer

The Critic estimates a value function to evaluate the Actor's actions and reduce gradient variance

Answer

The Critic learns a value function (V or Q) to assess how good the Actor's chosen actions are, providing a baseline that reduces variance in policy gradient estimates.

(AML) Advanced Machine Learning Certification Practice Test

(AML) Advanced Machine Learning Certification Practice Test

AML Reinforcement Learning & Decision Making 2