Machine Learning Reinforcement Learning 2 — Questions and Answers
Question 1: What is Q-learning in reinforcement learning?
- A clustering algorithm adapted for sequential environments
- A model-free RL algorithm that learns action-value (Q) functions (Correct answer)
- A method for dimensionality reduction in state spaces
- A supervised learning technique applied to sequential prediction
Correct answer: A model-free RL algorithm that learns action-value (Q) functions
Q-learning is a model-free RL algorithm that learns the value of taking action a in state s (Q-values) to find an optimal policy without requiring a model of the environment.
Question 2: What does the Bellman equation capture in reinforcement learning?
- The relationship between learning rate and convergence speed
- The optimal policy expressed as state transition probabilities
- The recursive relationship between the value of a state and its successor states (Correct answer)
- The formula for calculating policy entropy
Correct answer: The recursive relationship between the value of a state and its successor states
The Bellman equation expresses the value of a state as the immediate reward plus the discounted value of the next state, establishing recursive relationships used to solve RL problems.
Question 3: What is temporal difference (TD) learning?
- Learning from differences between consecutive time steps using bootstrapping before episode completion (Correct answer)
- A method for predicting future timestamps in time-series forecasting
- Training recurrent networks across multiple time periods simultaneously
- Measuring the delay between taking an action and receiving a reward
Correct answer: Learning from differences between consecutive time steps using bootstrapping before episode completion
TD learning updates value estimates based on the difference between predicted and observed rewards at consecutive time steps, combining ideas from Monte Carlo methods and dynamic programming.
Question 4: What is the key distinction between model-based and model-free reinforcement learning?
- Model-based uses neural networks while model-free uses lookup tables
- Model-based learns an environment model to plan ahead; model-free learns directly from interaction (Correct answer)
- Model-based requires more data while model-free requires less data to converge
- Model-based applies to discrete spaces while model-free applies to continuous spaces
Correct answer: Model-based learns an environment model to plan ahead; model-free learns directly from interaction
Model-based RL learns or uses a model of the environment's dynamics to plan ahead, while model-free RL learns policies or value functions directly from experience without building a model.
Question 5: In the SARSA algorithm, what does each letter represent?
- State, Action, Reward, Space, Algorithm
- State, Action, Reward, State, Action (Correct answer)
- Sample, Average, Reward, Step, Agent
- State, Agent, Return, Space, Action
Correct answer: State, Action, Reward, State, Action
SARSA stands for State, Action, Reward, State, Action — representing the tuple (s, a, r, s', a') used in its on-policy TD update rule.
Question 6: What is the purpose of the discount factor (γ) in reinforcement learning?
- To control the learning rate during gradient updates
- To determine how much future rewards are valued relative to immediate rewards (Correct answer)
- To reduce the exploration rate over time during training
- To normalize reward values to fall between 0 and 1
Correct answer: To determine how much future rewards are valued relative to immediate rewards
The discount factor γ (0 ≤ γ ≤ 1) determines how much the agent values future rewards relative to immediate rewards, with values closer to 0 making it more short-sighted.
Question 7: What is a Markov Decision Process (MDP) in reinforcement learning?
- A neural network architecture designed for sequential decision problems
- A mathematical framework for modeling sequential decision-making with states, actions, transitions, and rewards (Correct answer)
- A clustering algorithm for grouping similar time-series states
- A method for handling missing or delayed reward signals
Correct answer: A mathematical framework for modeling sequential decision-making with states, actions, transitions, and rewards
An MDP is a mathematical framework consisting of states, actions, transition probabilities, and rewards that formally models sequential decision-making under uncertainty.
Machine Learning Reinforcement Learning 2
What is Q-learning in reinforcement learning?