To decide what to do, the agent compares different sequences of rewards. The most common way to do this is to convert a sequence of rewards into a number called the value, the cumulative reward or the return. To do this, the agent combines an immediate reward with other rewards in the future. Suppose the agent receives the sequence of rewards:

Three common reward criteria are used to combine rewards into a value $V$:

Total reward

$V={\sum }_{i=1}^{\mathrm{\infty }}{r}_i$. In this case, the value is the sum of all of the rewards. This works when you can guarantee that the sum is finite; but if the sum is infinite, it does not give any opportunity to compare which sequence of rewards is preferable. For example, a sequence of $1 rewards has the same total as a sequence of $100 rewards (both are infinite). One case where the total reward is finite is when there are stopping states and the agent always has a non-zero probability of eventually entering a stopping state.

Average reward

$V={lim}_{n\to \mathrm{\infty }}(r_1+\mathrm{\cdots }+r_n)/n$. In this case, the agent’s value is the average of its rewards, averaged over for each time period. As long as the rewards are finite, this value will also be finite. However, whenever the total reward is finite, the average reward is zero, and so the average reward will fail to allow the agent to choose among different actions that each have a zero average reward. Under this criterion, the only thing that matters is where the agent ends up. Any finite sequence of bad actions does not affect the limit. For example, receiving $1,000,000 followed by rewards of $1 has the same average reward as receiving $0 followed by rewards of $1 (they both have an average reward of $1).

Discounted reward

$V=r_1+\gamma r_2+{\gamma }^2{r}_3+\mathrm{\cdots }+{\gamma }^{i-1}{r}_i+\mathrm{\cdots }$, where $\gamma$, the discount factor, is a number in the range . Under this criterion, future rewards are worth less than the current reward. If $\gamma$ was 1, this would be the same as the total reward. When $\gamma =0$, the agent ignores all future rewards. Having guarantees that, whenever the rewards are finite, the total value will also be finite.

The discounted reward can be rewritten as

	$V$	$={\displaystyle \sum _{i=1}^{\mathrm{\infty }}}{\gamma }^{i-1}{r}_i$
		$=r_1+\gamma r_2+{\gamma }^2{r}_3+\mathrm{\cdots }+{\gamma }^{i-1}{r}_i+\mathrm{\cdots }$
		$=r_1+\gamma (r_2+\gamma (r_3+\mathrm{\cdots })).$

Suppose $V_k$ is the reward accumulated from time $k$:

	$V_k$	$=r_k+\gamma (r_{k+1}+\gamma (r_{k+2}+\mathrm{\cdots }))$
		$=r_k+\gamma V_{k+1}.$

To understand the properties of $V_k$, suppose $S=1+\gamma +{\gamma }^2+{\gamma }^3+\mathrm{\cdots }$, then $S=1+\gamma S$. Solving for $S$ gives $S=1/(1-\gamma )$. Thus, with the discounted reward, the value of all of the future is at most $1/(1-\gamma )$ times as much as the maximum reward and at least $1/(1-\gamma )$ times as much as the minimum reward. Therefore, the eternity of time from now only has a finite value compared with the immediate reward, unlike the average reward, in which the immediate reward is dominated by the cumulative reward for the eternity of time.

In economics, $\gamma$ is related to the interest rate: getting $\$1$ now is equivalent to getting $\$(1+i)$ in one year, where $i$ is the interest rate. You could also see the discount rate as the probability that the agent survives; $\gamma$ can be seen as the probability that the agent keeps going.

The rest of this chapter considers discounted rewards. The discounted reward is referred to as the value.