The Bellman equation cannot be written recursively.

If , the agent only considers the immediate reward.

The optimal policy for a given task is always unique.

is a random variable representing the action taken at time .

In a discounted return, future rewards are multiplied by powers of .

The discount factor means the agent only cares about immediate rewards.

The discount factor satisfies .

The state-value function gives the expected number of actions before termination.

Imagine that we have the following first order Markov Process. In this figure, the numbers next to the arrows represent the transition probabilities .

We observe the sequence . What is the probability that is ?

If and , then .