The childcare benefits scandal and a recent case involving fraud checks by student finance provider DUO show that algorithms can have discriminatory effects. These cases are not isolated, says assistant professor in Mathematics and Computer Science Hilde Weerts: “You see it keeps happening.”

The Netherlands Institute for Human Rights (CRM) observes the same trend. It also notes that when discrimination stems from algorithms, it is difficult to pinpoint exactly where things go wrong.

Similarities

In recent years, fairness methods—mathematical techniques used to detect or even prevent discrimination in algorithms—have been used more frequently. However, according to CRM researcher and policy advisor Tim Peute it was unclear whether these methods sufficiently overlap with how discrimination is assessed in legal terms.

In the report Weerts wrote for the CRM, she concludes that while there are similarities, these methods are not capable enough of demonstrating or preventing discrimination.

For example, if an algorithm in a hiring process selects more men than women, fairness methods can measure that difference and its magnitude. However, they cannot determine whether there is a justified reason for that distinction, what the reference population should be, or what the consequences are for the disadvantaged women.

To determine whether the distinction made by an algorithm is justified, numbers alone are not enough, Weerts argues.

Prevention

According to Weerts, fairness methods aimed at preventing discrimination also often fall short because they focus too much on naïve optimization. In such cases, the algorithm is designed to perform as well as possible, with an added constraint—for example, that it must select an equal number of men and women. 

“But there are a billion ways to achieve that. And it’s not at all evident that those align with the reason you wanted equal selection of men and women in the first place.”

Completely preventing discrimination in algorithms is almost impossible, Weerts says. “That has to do with the technical side: how you build the algorithm, the data you have available—those kinds of factors. But also with the fact that you can only really determine discrimination once it has already occurred.”

The only way to prevent discrimination is to look at its deeper causes, she argues. “And to address those, rather than thinking afterward: we’ll just apply some optimization on top.”

Stopping altogether

If discrimination cannot be ruled out, should algorithms be used at all for applications such as fraud detection? “There are discussions about that, yes. I find it a difficult question,” says Weerts. “Amnesty International believes that the government has shown so clearly that it cannot do this properly, that it might be better to stop altogether. To some extent I agree, but I’m still an engineer at heart and think: if you really do it right, it should be possible in some cases.”

However, doing it “right” introduces another challenge: “If you look at how expensive it is to keep such an algorithm running, to fully develop it, and to involve all the necessary experts, you can question whether that is the most efficient way of operating. It might be better to focus on preventing people from committing fraud in the first place.”

For organizations that do want to use AI, Weerts’ report offers practical guidance. The CRM has published a document summarizing the findings and outlining what organizations can do, such as conducting analyses with a multidisciplinary team that includes at least a legal expert, a domain expert, and a data scientist.

Broader context

The report also helps the human rights institute in a broader context. The CRM is one of the fundamental rights authorities under the European AI regulation (also known as the AI Act), policy advisor Peute explains. This law sets requirements for organizations that develop and use AI systems. “AI providers, for example, must take measures themselves to detect, prevent, and mitigate bias in their data.” 

Thanks to Weerts’ report, the CRM now knows that assessing bias alone may not be sufficient to evaluate discrimination risks. “This means that a regulator should also consider a broader context in order to properly assess whether the risks of discrimination in AI use have been adequately addressed.”

Weerts herself is also involved in a broader initiative for fair AI in the Netherlands: the Dutch Technical Agreement for profiling algorithms. “Nearly thirty different organizations—including government institutions, NGOs, universities, and implementing agencies—have jointly written a guideline on how to properly implement fairness methods.”

The agreement is not binding, but it does provide guidance for organizations on how to apply profiling algorithms responsibly. And even here, fairness methods alone will not be sufficient. Anyone who wants to tackle discrimination in algorithms will need to take a much broader perspective.

This article was translated using AI-assisted tools and reviewed by an editor.