Preventing discrimination requires that we have means of detecting it, and this can be enormously difficult when human beings are making the underlying decisions. As applied today, algorithms can increase the risk of discrimination. But … algorithms by their nature require a far greater level of specificity than is usually possible with human decision making, and this specificity makes it possible to probe aspects of the decision in additional ways. With the right changes to legal and regulatory systems, algorithms can thus potentially make it easier to detect – and hence to help prevent – discrimination.

– Kleinberg et al. [2020]

Machine learning algorithms make predictions based on the selection of input features, the target, the data, and the evaluation criteria. Numerous machine learning models have been shown to incorporate systematic biases based on race, gender, and level of poverty. These can depend crucially on the input features and target. Obermeyer et al. [2019] report on a learned model for determining which patients will require more intensive care. Using historical data, the model predicted the costs of medical treatments, with the higher predicted costs getting more proactive treatment. They found that for the same symptoms, black people were systematically recommended to require less proactive treatment. This occurred because blacks have historically had less money spent on their healthcare. Even though the race was not an input feature, it was correlated with other features that were used in the prediction. While the amount on money spent may be an easy to measure proxy for health for those jurisdictions that track money, it does not correspond to the health, which is much harder to measure. Obermeyer et al. [2019] argued that the predictive model was flawed because the data was biased. In their case, they worked with the model designers to make the target more appropriate for the actual decision, and developed models that were much fairer.

Understanding the reasons behind predictions and actions is the subject of explainable AI. It might seem obvious that it is better if a system can explain its conclusion. Having a system that can explain an incorrect conclusion, particularly if the explanation is approximate, might do more harm than good.

There have been similar cases of biases in training data in many other domains, including models that predict crime as used in predictive policing [Lum and Isaac, 2016], and models of who to hire [Ajunwa, 2020]. The algorithms might have worked as intended, predicting patterns in the data to train them, but the data was biased. Datasets for facial recognition lead to models that are more prone to false positives for some populations than others [Buolamwini and Gebru, 2018], and mugshot datasets exacerbate over-surveillance of marginalized populations, perpetuating systemic racism.

People might legitimately disagree on the appropriate label for training data, such as toxicity and misinformation of social media posts. Different groups of people may have irreconcilable disagreements about the ground truth. Predicting the mode or average label can mean that minorities, who are often most at risk, are ignored [Gordon et al., 2021].

A model that performs well on held-out (validation) data might not work well in an application [Liao et al., 2021], due to internal or external validity. Internal validity refers to issues that arise in the learning problem in isolation, such as overfitting to standard validation sets – choosing the solutions that are best on validation or test sets – and using them for new problems. External validity refers to problems that arise on using the model for some task for which it might seem appropriate, such as due to differences in the dataset or the appropriate evaluation.

Data is invariably based on the past, but you might not want the future to be like the past. As Agrawal et al. [2022] argue, prediction is not an end in itself. Knowing predictions based on the past is useful, but does not directly specify what you should do. What an agent should do also depends on the values and preferences as well as what actions are available. How to make decisions based on predictions is the basis for much of the rest of the book.