…self-driving cars …The technology is essentially here. We have machines that can make a bunch of quick decisions that could drastically reduce traffic fatalities, drastically improve the efficiency of our transportation grid, and help solve things like carbon emissions that are causing the warming of the planet. But …what are the values that we’re going to embed in the cars? There are gonna be a bunch of choices that you have to make, the classic problem being: If the car is driving, you can swerve to avoid hitting a pedestrian, but then you might hit a wall and kill yourself. It’s a moral decision, and who’s setting up those rules?

– Barack Obama, 2016 [Dadich, 2016]

Sometimes agents face decisions where there are no good choices. They just have to choose which of the bad choices to carry out. A designer has to either explicitly or implicitly decide what the agents should do in such circumstances. Trolley problems consider hypothetical scenarios where a trolley-car (tram, streetcar) has brake failure and has to decide which of two tracks to go down. Both tracks have bad outcomes; for example, one may kill three people who are not supposed to be there, and the other may kill a worker doing their job. As the scenarios vary, people are asked which they prefer. A number of philosophers, following Foot [1967], have used such problems to probe people’s thinking about what to do when the interests of human beings conflict, including the difference between someone’s responsibility for harms that they cause (more or less directly) and for harms that they merely allow to happen.

In a modern variant of the trolley problem, the moral machines experiment asked millions of people from 233 countries about what autonomous vehicles (self-driving cars) should do in various circumstances.

The scenarios differed in the number of deaths, people versus animals, men versus women, young versus old, lawful versus unlawful, fit versus unfit. The global tendencies were to prefer sparing humans to animals, preference for sparing more lives, and preference for sparing young lives over old lives. Some preferences, such as the preference between genders and between social status (such as homeless person versus doctor), varied considerably between countries.

We can embrace the challenges of machine ethics as a unique opportunity to decide, as a community, what we believe to be right or wrong; and to make sure that machines, unlike humans, unerringly follow these moral preferences.

– Awad et al. [2018, p. 63]

This work showed that there are some principles that seem universal. There are some that are culturally specific. It also oversimplified in that it did not include any uncertainty; all of the outcomes were definitive. One interesting aspect is that, on average, people thought it was more important to save pedestrians than to save people in the vehicle; this means that it should not be up to the owners and drivers of the vehicles to choose their own policies, as people generally prefer to save their family than strangers.