Experts formulate “AI ground rules” for human supervision
Sarah Sterz © Saarland University/Thorsten Mohr
Artificial intelligence is already making far-reaching decisions today: Algorithms decide on loans, invitations to job interviews or show where a tumour is hiding. The EU has therefore passed the so-called “AI Act”, which defines how AI should be handled. According to this, “high-risk AI systems” should always be supervised by humans. However, it remains unclear exactly what this means. Experts from Saarland, Dresden and Freiburg have now provided a fundamental answer to this question.
If you enter the search term “AI fails” in the search engine of your choice, you get an idea of the scale of the problem: from funny little mistakes like “dogs in the NBA” to racist chatbots and stories about image recognition software that confuses humans with animals, it’s all there. It quickly becomes clear: AI makes mistakes. And sometimes very serious ones.
Artificial intelligence is already making far-reaching decisions that can have a massive impact on people’s lives. AI systems are used, for example, in lending, in application procedures and in medicine, to name just a few examples. Depending on the case, the person seeking a loan will either receive no offer at all, an expensive loan or a favourable one. In the application process, AI filters out unsuitable applicants. And AI-supported image processing software helps doctors, for example, to find a tumour in the body.
To ensure that the decisions made by such AI systems are as fair as possible, the European Commission recently passed the “AI Act”. It stipulates that such “high-risk AI systems”, which can have a major impact on people’s lives, can also be “effectively supervised” by humans. However, the legislator has not specified exactly what this means, i.e. what distinguishes effective from ineffective human supervision. A group of scientists from the fields of computer science, philosophy, psychology and law from all over Germany (Saarbrücken, Dresden and Freiburg) have tackled this complex question and have now presented a set of criteria intended to provide developers and users of AI systems as well as legislators and courts with a framework to ensure effective supervision.
“Basically, we have defined four criteria that a human must fulfil in order to ensure this ‘effective oversight’ of an AI system,” says first author Sarah Sterz, explaining the core of the highly acclaimed paper. It was published in early June at the ACM Conference on Fairness, Accountability and Transparency (ACM FAccT). “This is the conference for such topics worldwide,” adds Kevin Baum. The head of the Center for European Research in Trusted Artificial Intelligence (CERTAIN) at the German Research Centre for Artificial Intelligence (DFKI) in Saarbrücken is co-author of the paper.
“Firstly, a human supervisor must have causal effectiveness over the system,” Sarah Sterz begins the list with the first of the four defined framework conditions. “In concrete terms, this means that the human must be able to intervene in the system and override its decision.” In practice, this could be an emergency stop button on the robot that helps in the industrial hall, or the ability to overrule an AI that decides who is invited and who is not in the application process.
“Secondly, the human supervisor must know how the system works and what the consequences of their own interventions would be. They must have an epistemic understanding of the AI system and their own options for action,” Sarah Sterz continues.
“Thirdly, the person must also have sufficient self-control to supervise an AI effectively,” explains Sarah Sterz. The person must be mentally and physically capable of fulfilling this task. “For example, you shouldn’t be drunk, overtired or bored,” says the computer scientist, giving three examples. “If you have already sifted through 200 applications in one day, you might make mistakes with the 201st application. According to this criterion, they would no longer be able to effectively supervise an AI system,” says the scientist, who is doing her doctorate under computer science professor Holger Hermanns at Saarland University, giving a practical example of this criterion.
Last but not least, they should have the right intentions, according to the fourth criterion defined by Sarah Sterz and her co-authors. “Super villains, for example, would therefore be unsuitable per se to supervise AI systems, even if they understand how they work, have concrete causal power over them and have sufficient self-control,” says Sarah Sterz, giving a striking example. A film villain à la Dr Evil from “Austin Powers” would therefore be unsuitable to supervise an AI, as it would not be in his nature to eliminate the risks of the AI in favour of the people over whom it makes decisions. This would be the case even if he were well-rested, sober and tech-savvy enough to know how the AI works, i.e. if he met the other criteria. However, it is not only supervillains but also unmotivated staff that could become a problem, because anyone who is not willing to avert risks will not be able to supervise a system effectively.
Computer scientist and philosopher Kevin Baum explains why such rules are important: “AI systems will simply always make mistakes, no matter how far technology advances. Worse still, it is generally mathematically impossible to design an AI system that makes the same mistakes with the same probability for everyone. That’s why a framework like the one we’ve now created is so important, to allow human supervisors to detect errors as early as possible and invalidate malicious outputs. With our criteria, we want to offer a structure, a framework to make this possible,” he explains.
The complexity of this issue can be recognised by the large number of scientific disciplines involved. “The question of human oversight of AI is not a purely legal question, it is not a purely philosophical question, it is not a purely psychological question and it is not a purely information technology question. Rather, it is a question whose answer must combine the perspectives of all these disciplines,” explains Kevin Baum.
In the meantime, the EU Commission has answered the question itself: Sarah Sterz, Kevin Baum and their colleagues have been invited by the European Office for Artificial Intelligence to a conference in Sweden in September to explain their policy paper to the participants there.
Original publication:
Sarah Sterz, Kevin Baum, Sebastian Biewer, Holger Hermanns, Anne Lauber-Rönsberg, Philip Meinel, and Markus Langer. 2024. On the Quest for Effectiveness in Human Oversight: Interdisciplinary Perspectives. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’24). Association for Computing Machinery, New York, NY, USA, 2495–2507. https://doi.org/10.1145/3630106.3659051
The work was carried out as part of the Transregional Collaborative Research Centre “TRR 248: Fundamentals of Comprehensible Software Systems – for a Comprehensible Cyber-Physical World”, which is funded by the German Research Foundation.
Further information:
Sarah Sterz
Phone: (0681) 3025589
Mail: sterz(at)depend.uni-saarland.de
Kevin Baum
Phone: (0681) 857755251
Mobile: (0151) 530513 84
Mail: kevin.baum(at)dfki.de
Background German Research Center for Artifical Intelligence:
The German Research Center for Artificial Intelligence (DFKI) has operated as a non-profit, Public-Private-Partnership (PPP) since 1988. Today, it maintains sites in Kaiserslautern, Saarbrücken, Bremen, Niedersachsen, laboratories in Berlin, Darmstadt and Lübeck as well as branches in Trier. DFKI combines scientific excellence and commercially-oriented value creation with social awareness and is recognized as a major “Center of Excellence” by the international scientific community. In the field of artificial intelligence, DFKI has focused on the goal of human-centric AI for more than 35 years. Research is committed to essential, future-oriented areas of application and socially relevant topics. Currently, with a staff of about 1,560 employees from more than 76 countries, DFKI is developing the innovative software technologies of tomorrow.
Background Saarland Informatics Campus:
900 scientists (including 400 PhD students) and approx. 2500 students from more than 80 nations make the Saarland Informatics Campus (SIC) one of the leading locations for computer science in Germany and Europe. Four world-renowned research institutes, namely the German Research Center for Artificial Intelligence (DFKI), the Max Planck Institute for Informatics, the Max Planck Institute for Software Systems, and the Center for Bioinformatics along with Saarland University and its three departments and 24 degree programs, together cover the entire spectrum of computer science.
Originally published at:
