Reinforcement Learning Flashcards

What is the key innovation of Deep Q-Networks (DQN) over classic Q-learning?

Answer: Using a deep neural network to approximate Q-values for high-dimensional state spaces

DQN uses a deep neural network to approximate Q-values, enabling RL to scale to high-dimensional state spaces such as raw pixel inputs from Atari games.

What do policy gradient methods optimize?

Answer: Policy parameters by computing gradients of the expected cumulative reward

Policy gradient methods directly optimize the policy parameters by computing gradients of expected cumulative reward with respect to those parameters.

What is the primary purpose of experience replay in DQN?

Answer: To store past transitions and sample them randomly to break temporal correlations and stabilize training

Experience replay stores past transitions in a replay buffer and randomly samples mini-batches for training, breaking temporal correlations and improving data efficiency.

What does the Actor-Critic method combine in reinforcement learning?

Answer: A policy network (actor) that selects actions combined with a value network (critic) that evaluates them

Actor-Critic methods combine an actor (policy network) that selects actions with a critic (value network) that evaluates those actions, reducing variance compared to pure policy gradient methods.

What is the epsilon-greedy strategy used for in reinforcement learning?

Answer: To balance exploration and exploitation by taking random actions with probability epsilon

The epsilon-greedy strategy selects a random action with probability epsilon (exploration) and the greedy best-known action with probability 1-epsilon (exploitation), balancing discovery and performance.

How does the Monte Carlo method estimate value functions in reinforcement learning?

Answer: By averaging actual returns from complete episodes after visiting each state

Monte Carlo methods in RL learn value functions by averaging the actual returns experienced after visiting each state across complete episodes, without bootstrapping.

What is the 'credit assignment problem' in reinforcement learning?

Answer: The challenge of determining which past actions were responsible for a delayed reward

The credit assignment problem is the challenge of identifying which actions in a sequence were responsible for a reward that arrived much later, since rewards are often delayed and sparse.