Open and accountable decision making requires making utilities explicit, where they are open to scrutiny and people can argue why they are appropriate or not; see the box. When a decision is proposed, sensitivity analysis – exploring how the decision changes as utilities change – can determine just how much the utility matters. Making utilities explicit and open is particularly important for decisions in the public sphere that affect many people, however these are often controversial because people do not have a common utility function.

Designing utilities and rewards so that the policies have desirable properties is called utility engineering or reward engineering. This is particularly difficult for unobservable constructs such as socioeconomic status, teacher effectiveness, and risk of recidivism for decisions on poverty reduction, education, or crime. These cannot be measured directly, but need to be inferred from a measurement model of observable properties, such as death rate, student ratings, or rearrests. However, an agent optimizing for the measurement model might not actually optimize for what is desired. Jacobs and Wallach [2021] analyze the interaction of measurement models and fairness.

One case where explicit utilities have been used is for resource allocation in health care, particularly for jurisdictions where the allocation is based on need, not ability to pay. A measure used in a number of countries is the quality-adjusted life year (QALY), a utility-based measure for evaluating medical interventions, such as (expensive) drugs or surgeries. For each possible intervention, it uses a utility of 1 for a healthy life for a year and 0 for death. The utility can be negative for outcomes that are considered worse than death. Outcomes are assessed as a lottery between the maximum and minimum utilities. The utility of the intervention is the sum of this value over each expected year of life. If the intervention is ongoing and constant in the future, the utility is the life expectancy times the yearly value. The QALY provides a measure that incorporates the quantity and quality of life. When there are limited resources, the cost/QALY ratio is used as a cost-effectiveness measure for decision making in many countries.

Society must make decisions that affect everyone. Finding a utility that works for everyone is controversial. For example, for most sighted people, going blind would have a low utility; they consider going blind to be very bad as they would need to relearn the way they interact with the world. So the utility for blindness for a year would be low. However, this implies that blind people are less valued than sighted people, an ableist assumption. There have been suggestions for incorporating individual reference points, as in prospect theory.

Sometimes society needs to make life-and-death decisions. For example, consider how much to spend on earthquake-proofing public schools. A severe earthquake when pupils are in school might cause multiple deaths. It is possible to compute the probability of an earthquake in a location and the probability that a particular structure will collapse when students are present. Money can be spent to reduce the chance of a collapse. Deciding whether to spend the money is a classic example of decision making under uncertainty, which requires trading off money with children’s lives. Many decision makers are reluctant to explicitly trade off money and the lives of children. However, when they don’t make an explicit trade-off they tend to undervalue children’s lives.

As AI tools become more common, and more decisions are automated, society will need to come to terms with the uncomfortable conversations of assigning utilities. Not having these conversations will mean that someone else’s utilities are embedded in AI tools.