Dagstuhl Seminar 24182
Resilience and Antifragility of Autonomous Systems
( Apr 28 – May 03, 2024 )
(Click in the middle of the image to enlarge)
Permalink
Please use the following short url to reference this page:
https://www.dagstuhl.de/24182
Organizers
- Simon Burton (Gerlingen, DE)
- Radu Calinescu (University of York, GB)
- Raffaela Mirandola (KIT - Karlsruher Institut für Technologie, DE)
Contact
- Michael Gerke (for scientific matters)
- Christina Schwarz (for administrative matters)
Shared Documents
- Dagstuhl Materials Page (Use personal credentials as created in DOOR to log in)
Schedule
The increasing complexity in the environment, tasks, and technology related to autonomous systems results in limitations in the statements that can be made regarding dependability during design time. In particular, these systems may operate within environments for which only incomplete models exist, that may change over time or may be subject to unforeseen interactions and disturbances. As a result, such systems must be engineered to be trustworthy despite residual insufficiencies in their design, and in the presence of unexpected events due to their dynamically evolving operating context.
Related domains concerned with system autonomy in uncertain environments have already taken inspiration from nature to endow artificial systems with self-* properties (e.g. self- optimisation, -repair, -protection, -configuration, and -adaptation). Such self-* capabilities enable systems to improve their performance and dependability at runtime while reducing the need for low-level human intervention -- properties that are closely related to resilience and antifragility.
This Dagstuhl Seminar aimed to unify the international research on resilient and antifragile autonomous systems (RAAS), leading to faster scientific advancements and industrial adoption. To this end, the seminar brought together leading researchers and practitioners with expertise in autonomous system resilience, antifragility, safety, and ethics, from disciplines including computer science, safety science, and ethics, to share and discuss each other's understanding of, methods for, and open challenges related to RAAS. Initial presentations were used to set the scene by proposing basic definitions, industry perspectives, and engineering views on cyber-resilience. These were followed by group and plenary discussions to explore these concepts in more detail.
A clear set of agreed definitions is essential in order to make progress as a community in this area. Resilience can be broadly seen as the ability to absorb disturbances and unexpected events whilst maintaining essential properties of the system. Using such conditions to harden the system against future events can be viewed as antifragility. These definitions highlight that antifragility is a concept referring to systems designed to operate under "open-world" assumptions, where the responsibility of maintaining a given property, despite disturbances (resilience) mostly shifts from design time to runtime, and relies on the presence in the system of some suitable degree of autonomy (self-* capability). As such, antifragility can be viewed as the ability of a system to self-improve its resilience over (run)time. Discussions converged to the idea that in order to define resilience and antifragility, we should build on the work of Control Theory, specifically how systems recover from (potentially previously unknown) disturbances. Thus, we postulated that both resilience and antifragility should be defined over the metrics of settling time, percentage of settling, percentage of overshoot, and percentage of overshoot with respect to the properties of interest in the event of disturbances to the system. Discussions on how to use formal methods to construct systems that guarantee these desired properties generated many challenging questions that are to be followed up in future research.
Initial work in the seminar explored more precise definitions of RAAS that also included the consideration of uncertainty and causality, and where a collection of properties may need to be optimised as a whole. Such trade-offs are particularly evident when considering safety, ethical, and legal aspects of RAAS. In some cases, autonomous systems must remain operational in order to stay safe. A resilient system could remain within its safety bounds when disrupted, whilst maintaining a minimal level of utility. An antifragile system could use repeated disturbances to lower risk over time whilst increasing overall utility. Similar trade-offs and optimisations will be found when considering legal and ethical concerns for RAAS and these could lead to specific technical requirements on the system. For example, for a system that adapts its function over time, avoiding the loss of agency in human stakeholders needs to be ensured.
Engineering antifragile systems requires specialised consideration in each phase of the traditional software and system development process. This includes requirements, design, implementation, and testing. Artificial Intelligence (AI) - in terms of machine learning, symbolic AI techniques, and combinations thereof - has the potential to provide a basis for both recognising disturbances and deciding the system adaptations needed to mitigate these disturbances. The seminar participants see potential for AI to be used in all phases of the MAPE-K (monitor-analyse-plan-execute supported by knowledge) cycle of self-adaptive systems. Furthermore, a control-theoretic reasoning approach could be used to verify whether a particular adaptation manager pushes the resilience error (i.e., the difference between observed and preferred resilience) below some threshold, or whether the resilience level stabilises at a reference value.
The seminar concluded that much work is still required to advance research in the area of RAAS, and to foster RAAS adoption in industrial applications. This includes:
- Agreeing on terminology and definitions that build upon and extend our traditional understanding of dependable systems;
- Formally defining metrics for resilience and antifragility that can be used to design and verify RAAS;
- Engineering methods and candidate technologies for implementing RAAS;
- Considering the safety, legal, and ethical implications of RAAS, including both their positive potential and their associated risks.
The participants agreed to pursue these important and challenging issues in future collaborations, including joint publications, workshops, and journal special issues.
Creative Commons BY 4.0
Simon Burton, Radu Calinescu, and Raffaela Mirandola
Autonomous systems are developed at a staggering pace for use in healthcare, transportation, manufacturing, and many other domains. Often comprising artificial intelligence (AI) components, these systems are uniquely capable of performing tasks and making decisions in changing environments, and can operate without human intervention for extended periods of time. As such, autonomous systems have the potential to undertake or support complex missions that are dangerous, difficult or tedious for humans.
To achieve this potential, autonomous systems must be resilient, i.e., they must continue to provide the required functionality despite the uncertainty, change, faults, failure, adversity, and other (anticipated and unforeseen) disruptions within their operating environments. Numerous methods for developing resilient autonomous systems have been proposed, including: (1) design methods for developing autonomous systems that resist disruption through robustness and fault tolerance; (2) reactive adaptation methods which ensure that autonomous systems absorb disruption, e.g., by graceful degradation; and (3) proactive adaptation methods that anticipate disruption by recognizing patterns of evolution, and avoiding disruption proactively. Additional research has been devoted to ensuring that these methods can be used without compromising the safety or violating the ethical norms and rules of system users and operators.
The ability of autonomous systems to achieve their goals in open real-world environments can be further increased by making them antifragile. Antifragile systems benefit from exposure to uncertainty and disruption, by learning from encounters with such difficulties, so that they can handle their future occurrences faster, more efficiently, with lower user impact, etc. The inspiration for antifragility comes from nature, where antifragile systems are ubiquitous. For example, the immune system responds to exposure to pathogens by producing antibodies that help protect against future infections.
Despite recent advances in autonomous technologies, the research on resilient autonomous systems remains fragmented and lacks industrial adoption, and that on antifragile autonomous systems is in its infancy. As these closely related research areas play a key role in the realization of the societal and economic benefits of autonomous systems, now is the right time for the international research communities from these areas to come together, to identify synergies across their disciplines and research programmes, and to agree on a common basis for joint future research.
This Dagstuhl Seminar aims to unify the international research on resilient and antifragile autonomous systems (RAAS), leading to faster scientific advancements and industrial adoption. To that end, the seminar wants to bring together leading researchers and practitioners with expertise in autonomous system resilience, antifragility, safety, and ethics from disciplines including Computer Science, Computational Biology, and Ethics, to share and discuss each other’s understanding of, methods for, and open challenges related to RAAS. These participants will work closely together to: (1) survey the current RAAS research in order to develop and document a common understanding of the RAAS research landscape; (2) identify RAAS open challenges and promising preliminary approaches to tackling them; (3) set an international research agenda for addressing these challenges; (4) define a roadmap for the delivery of this agenda; and (5) agree on use cases (e.g., from health and assistive care, transportation, aviation and aerospace) that can be used as a benchmark for the evaluation of future RAAS solutions.
The seminar and its outputs will cover a broad range of RAAS topics. These topics will include: (1) RAAS concepts, terminology and measures; (2) state-of-the-art methods for autonomous system resilience and their integration; (3) nature-inspired approaches to autonomous system resilience and antifragility; and (4) safety and ethical concerns of RAAS.
Simon Burton, Radu Calinescu, Raffaela Mirandola, and Corina Pasareanu
Please log in to DOOR to see more details.
- Lee Barford (Keysight Technologies - London, GB)
- Amel Bennaceur (The Open University - Milton Keynes, GB) [dblp]
- Simon Burton (Gerlingen, DE) [dblp]
- Radu Calinescu (University of York, GB) [dblp]
- Matteo Camilli (Polytechnic University of Milan, IT)
- Marc Carwehl (HU Berlin, DE) [dblp]
- Ana Cavalcanti (University of York, GB) [dblp]
- Felicita Di Giandomenico (CNR - Pisa, IT) [dblp]
- Ada Diaconescu (Telecom Paris, FR) [dblp]
- Kerstin I. Eder (University of Bristol, GB) [dblp]
- Mario Gleirscher (Universität Bremen, DE) [dblp]
- Vincenzo Grassi (University of Rome "Tor Vergata", IT) [dblp]
- Sebastian Hahner (KIT - Karlsruher Institut für Technologie, DE) [dblp]
- Andreas Heyl (Robert Bosch GmbH - Stuttgart, DE)
- Antje Loyal (Continental Automotive Technologies - Frankfurt, DE)
- Ravi Mangal (Carnegie Mellon University - Pittsburgh, US)
- Lina Marsso (University of Toronto, CA)
- Raffaela Mirandola (KIT - Karlsruher Institut für Technologie, DE) [dblp]
- Gabriel Moreno (Carnegie Mellon University - Pittsburgh, US)
- Elena Navarro (University of Castilla-La Mancha, ES)
- Shiva Nejati (University of Ottawa, CA) [dblp]
- Diego Perez-Palacin (Linnaeus University - Växjö, SE) [dblp]
- Ralf H. Reussner (KIT - Karlsruher Institut für Technologie, DE) [dblp]
- Patrizia Scandurra (University of Bergamo - Dalmine, IT) [dblp]
- Catia Trubiani (Gran Sasso Science Institute - L'Aquila, IT) [dblp]
- Sebastián Uchitel (University of Buenos Aires, AR)
- Gricel Nidteja Vazquez Flores (University of York, GB)
Classification
- Artificial Intelligence
- Computers and Society
- Systems and Control
Keywords
- resilience
- antifragility
- autonomous systems
- AI
- safety
- ethics and assurance of autonomous systems
