AML Reinforcement Learning & Decision Making 2 — Questions and Answers
Question 1: In Q-learning, what does the Q-value Q(s, a) represent?
- The probability of selecting action a when in state s
- The expected cumulative discounted reward when taking action a in state s and following the optimal policy thereafter (Correct answer)
- The immediate reward received after taking action a in state s
- The number of times action a has been taken in state s
Correct answer: The expected cumulative discounted reward when taking action a in state s and following the optimal policy thereafter
Q(s, a) is the expected total discounted return from taking action a in state s and then acting optimally, combining immediate reward with future expected returns.
Question 2: What fundamentally distinguishes SARSA from Q-learning?
- SARSA uses neural networks while Q-learning uses lookup tables
- SARSA is on-policy and updates using the actual next action taken; Q-learning is off-policy and updates using the greedy max-Q action (Correct answer)
- SARSA requires an environment model while Q-learning does not
- SARSA targets continuous action spaces while Q-learning targets discrete ones
Correct answer: SARSA is on-policy and updates using the actual next action taken; Q-learning is off-policy and updates using the greedy max-Q action
SARSA (on-policy) updates Q-values using the action actually taken by the behavior policy, while Q-learning (off-policy) always bootstraps from the greedy maximum Q-value action.
Question 3: What is the primary motivation for using Deep Q-Networks (DQN) over tabular Q-learning?
- DQN converges faster in all environments regardless of state space size
- DQN uses neural networks to approximate Q-values, enabling generalization over large or continuous state spaces (Correct answer)
- DQN eliminates the need for a replay buffer during training
- DQN removes the requirement for a discount factor
Correct answer: DQN uses neural networks to approximate Q-values, enabling generalization over large or continuous state spaces
DQN approximates Q-values with neural networks, enabling generalization across similar states and making RL tractable in high-dimensional or continuous state spaces where a Q-table is infeasible.
Question 4: What problem does experience replay solve in Deep Q-Networks?
- It prevents catastrophic forgetting of previously learned tasks
- It reduces correlation between consecutive training samples, stabilizing neural network learning (Correct answer)
- It increases the exploration rate during early training stages
- It allows DQN to train simultaneously across multiple environments
Correct answer: It reduces correlation between consecutive training samples, stabilizing neural network learning
Experience replay stores transitions in a buffer and randomly samples mini-batches, breaking the temporal correlation between consecutive samples that would otherwise cause unstable, divergent training.
Question 5: In the Actor-Critic architecture, what is the primary role of the Critic?
- The Critic selects and executes actions in the environment
- The Critic estimates a value function to evaluate the Actor's actions and reduce gradient variance (Correct answer)
- The Critic explores the environment by generating random action sequences
- The Critic directly computes the policy gradient
Correct answer: The Critic estimates a value function to evaluate the Actor's actions and reduce gradient variance
The Critic learns a value function (V or Q) to assess how good the Actor's chosen actions are, providing a baseline that reduces variance in policy gradient estimates.
Question 6: What is the key innovation that makes Proximal Policy Optimization (PPO) stable during training?
- PPO uses Monte Carlo returns exclusively instead of TD estimates
- PPO clips the policy update ratio to prevent excessively large policy changes that could destabilize learning (Correct answer)
- PPO requires a separate world model for environment planning
- PPO eliminates the value function baseline to simplify optimization
Correct answer: PPO clips the policy update ratio to prevent excessively large policy changes that could destabilize learning
PPO constrains updates by clipping the probability ratio between the new and old policy within a trust region, preventing destructively large policy updates while keeping training computationally efficient.
Question 7: What is the 'deadly triad' in reinforcement learning, and why is it problematic?
- Exploration, exploitation, and generalization — they create oscillating reward signals
- Function approximation, bootstrapping, and off-policy training — their combination can cause divergence and instability (Correct answer)
- Reward, policy, and value function — optimizing all three simultaneously leads to conflicting gradients
- Speed, accuracy, and memory — hardware constraints prevent jointly optimizing all three
Correct answer: Function approximation, bootstrapping, and off-policy training — their combination can cause divergence and instability
The deadly triad describes how combining function approximation, bootstrapping (TD updates), and off-policy learning introduces compounding approximation errors that can cause Q-value estimates to diverge.
AML Reinforcement Learning & Decision Making 2
In Q-learning, what does the Q-value Q(s, a) represent?