This Dagstuhl Seminar will investigate challenges and approaches to designing Artificial Intelligence (AI) systems that ensure meaningful human oversight and control of their operation.

The past decade has seen a substantial leap forward in AI research and technology, spanning from novel neural network-based deep learning approaches to applications of generative AI based on large language models. Rapidly, these technologies are being applied to numerous fields. Especially the deployment of AI-based systems in high-risk contexts entails threats to safety and fundamental human rights. To alleviate such risks, emerging ethical guidelines and legislation (such as the European AI Act) around the globe are calling for Human Oversight of AI-based systems in high-risk contexts.

However, the conditions for effective human oversight are ill-understood and so is their technological basis. Among the urgent questions that developers and deployers of AI-based systems have to deal with once such AI regulations are in force include: How to design interfaces and the communication between humans and systems to enable overseers to effectively control AI-based systems? Can explainability and visualization approaches promote an understanding of system capacities and limitations? How do we ensure that people override system outputs in situations where this reduces risks and does not introduce new risks?

This urgently calls for an interdisciplinary discussion among researchers in artificial intelligence, computer system design and verification, human-computer interaction, psychology, ethics, and law. This seminar involves AI systems researchers who need to work on ensuring system interpretability and advancing explainability approaches in a way that serves the needs of human overseers. This involves formal methods researchers who will help to ensure the accuracy and reliability of explanations stemming from approaches in Explainable Artificial Intelligence (XAI) research. This involves language processing and visualization researchers to effectively communicate information to human overseers enabling them, for instance, to effectively grasp current system states and risky situations. And this critically involves human-computer interaction researchers , for designing human-system decision workflows and oversight support tools that are tailored to the tasks of human overseers.

Apart from computer science experts, the seminar needs the perspective of psychology to understand people’s needs in ensuring effective human oversight, and in developing evaluation methods to empirically test whether human oversight is truly effective. We also need the perspectives of law and normative ethics to interpret the foundational assumptions behind emerging regulation and its connections to other legal frameworks. And eventually, we need all perspectives to respond to the intertwined questions: How to assess whether we have achieved effective human oversight? What are the requirements for “overseeability-by-design”? What if we find that there are limits to human oversight? How can and perhaps even should we get involved in shaping policy-making to take into account the limits and conditions of effective human oversight?

Among the major topics to be discussed during the seminar are: the scope and requirements of human oversight, AI systems built for human oversight, effective and intuitive user interfaces for human oversight, the evaluation of human oversight approaches, decision-making and associated challenges, and how to interface to law, norms, and society.

Creative Commons BY 4.0

Raimund Dachselt, Markus Langer, Q. Vera Liao, Timothy Miller, and Nava Tintarev