Artificial Intelligence Reinforcement Learning 2 — Questions and Answers
Question 1: What is Q-learning in reinforcement learning?
- A policy gradient algorithm that optimizes the policy directly
- A model-based planning algorithm using a learned transition model
- A model-free algorithm that learns action-value (Q) functions to determine the best action in each state (Correct answer)
- A supervised learning algorithm adapted for sequential decisions
Correct answer: A model-free algorithm that learns action-value (Q) functions to determine the best action in each state
Q-learning learns Q(s,a) — the expected cumulative discounted reward of taking action a in state s — and derives the optimal greedy policy from these values.
Question 2: What does the discount factor (γ) control in reinforcement learning?
- The learning rate of the value network
- The importance of future rewards relative to immediate rewards; values near 1 make the agent far-sighted (Correct answer)
- The probability of a random action under epsilon-greedy exploration
- The maximum number of steps per episode
Correct answer: The importance of future rewards relative to immediate rewards; values near 1 make the agent far-sighted
A discount factor γ ∈ [0,1) exponentially reduces the weight of future rewards; γ near 0 makes the agent myopic (focus on immediate reward), while γ near 1 makes it long-sighted.
Question 3: What is the Bellman equation used for in reinforcement learning?
- Computing the gradient of the policy network
- Expressing the value of a state as the immediate reward plus the discounted value of the next state, forming the basis for dynamic programming (Correct answer)
- Measuring the entropy of the policy distribution
- Defining the structure of the neural network used as a function approximator
Correct answer: Expressing the value of a state as the immediate reward plus the discounted value of the next state, forming the basis for dynamic programming
The Bellman equation recursively decomposes the value function: V(s) = max_a [R(s,a) + γ Σ P(s'|s,a) V(s')], enabling iterative computation of optimal values.
Question 4: What is Deep Q-Network (DQN) and what was its breakthrough application?
- A convolutional network for image segmentation applied to robotics
- A deep RL algorithm that uses a neural network to approximate Q-values, first achieving human-level performance on Atari games (Correct answer)
- A language model fine-tuned with human feedback for dialogue
- A multi-agent algorithm for simulated trading
Correct answer: A deep RL algorithm that uses a neural network to approximate Q-values, first achieving human-level performance on Atari games
DQN (DeepMind, 2015) uses a deep CNN to approximate Q(s,a) directly from raw pixel inputs, using experience replay and target networks to stabilize training on Atari 2600 games.
Question 5: What is 'experience replay' in DQN and why is it important?
- Replaying successful episodes to the agent as demonstrations
- Storing past transitions in a buffer and sampling random mini-batches to break temporal correlations and stabilize training (Correct answer)
- Replaying the training process from a saved checkpoint
- Re-running episodes in a simulated environment faster than real time
Correct answer: Storing past transitions in a buffer and sampling random mini-batches to break temporal correlations and stabilize training
Experience replay stores (s, a, r, s') transitions in a replay buffer and samples random mini-batches, decorrelating consecutive training samples and improving data efficiency.
Question 6: What is the key idea behind policy gradient methods in reinforcement learning?
- Learning a Q-value function and deriving the policy from it
- Directly optimizing the parameters of a policy by computing gradients that increase the probability of high-reward actions (Correct answer)
- Using dynamic programming to compute optimal values
- Building a model of the environment to plan ahead
Correct answer: Directly optimizing the parameters of a policy by computing gradients that increase the probability of high-reward actions
Policy gradient methods parameterize the policy directly and update parameters by ascending the gradient of expected reward, computed via the log-probability of taken actions weighted by their returns.
Artificial Intelligence Reinforcement Learning 2
What is Q-learning in reinforcement learning?