Dimensions: flat, features, finite horizon, partially observable, stochastic, utility, non-learning, single agent, offline, perfect rationality

Generally, agents do not make decisions in the dark without observing something about the world, nor do they make just a single decision. A more typical scenario is that the agent makes an observation, decides on an action, carries out that action, makes observations in the resulting world, then makes another decision conditioned on the observations, and so on. Subsequent actions can depend on what is observed, and what is observed can depend on previous actions. In this scenario, it is often the case that the sole reason for carrying out an action is to provide information for future actions. Actions that are carried out to just acquire information are called information seeking actions. Such actions are only ever needed in partially observable environments. The formalism does not need to distinguish information seeking actions from other actions. Typically actions will have both information outcomes as well as effects on the world.

A sequential decision problem models

• •

  what actions are available to the agent at each stage

• •

  what information is, or will be, available to the agent when it has to act

• •

  the effects of the actions and

• •

  the desirability of these effects.