(0, 1.75)

(1.25, 0)

(1.75, 1.85)

(1.0, 1.0)

(0, 0)

(0.5, 0)

(1.5, 1.75)

(0, 1.0)

SARSA

Monte Carlo

ε-greedy TD(0)

Expected SARSA

Q-Learning

TD(0)

 SARSA requires a model of the environment, while Q-Learning does not

Q-Learning updates only at the end of an episode, while SARSA updates at each step

Q-Learning is on-policy, while SARSA is off-policy

SARSA updates the Q-value using the actual action taken, while Q-Learning updates using the maximum action-value

TD methods use sample updates, DP methods use expected updates.

TD methods use expected updates, DP methods use sample updates.

Both TD and DP methods use expected updates.

Both TD and DP methods use sample updates.

Both DP and TD require a complete model of the environment’s dynamics.

The number of policies being evaluated

The number of future time steps used to compute the return

The number of actions taken in episode

The number of episodes to average over

The sum of rewards from step t to the end of the episode

The current estimated value of the state

The maximum Q-value over all actions

The discounted sum of the next n rewards and the estimated value of the nth state