Many of the Human Factors that impact how people program are poorly understood, presenting significant challenges to efforts aimed at improving programmer productivity and education. Programming error messages (commonly called compiler error messages) are one aspect that is known to be problematic, with evidence dating back to the 1960s. The information that is presented to users via these messages impacts different people in different ways, from children to students to seasoned professionals. Despite clear evidence of their human impact, no consensus exists on how to assess the effectiveness of error messages, how to improve them with large, demonstrated effect, nor how to produce appropriate messages for different users with different needs.

Work on improving programming error messages presents a complex challenge as it spans many sub-disciplines of Computer Science including Programming Languages, HCI, Computer Science Education, and Software Engineering as well as Learning Sciences more broadly. The past several years has seen a marked increase in interest within some communities. However, in order to best proceed it would be highly beneficial for experts across these areas to identify gaps in knowledge, high-priority areas, and a basis upon which future work can be built. This Dagstuhl seminar will seek to establish a long-lasting contribution to the field by uniting communities that have to date been working largely in isolation. We seek to gather appropriate and useful data, broaden perspectives, build consensus among diverse stakeholders, and begin cross-community efforts working in unison going forward.

During this Dagstuhl Seminar, participants will engage in a variety of activities to meet three broad objectives. First, we will have sessions focusing on existing literature and practice, including experience reports from language maintainers, expert users, researchers, and educators, to develop a shared understanding of the evidence that exists with regard to effective programming error messages. This will include the group working together to synthesize existing data sets, initially contributed by attendees with access to such. This is an important exercise, as large corpora of errors and error message data are generally not openly available, making it difficult for researchers to answer seemingly simple questions (without conducting large scale studies) such as: "What are the most frequent error messages encountered by [students, professionals, blind, non-native English speakers] in language x?" or, "What are evidence-based examples of effective and/or ineffective error messages?" Organizing such data will help meet the second objective of the seminar: to identify areas for immediate research including determining the design factors of programming error messages that differ across demographic groups (e.g., expertise level, disabilities, native/natural language, etc.). The third objective of the seminar is to establish new cross-community efforts to improve programming error messages in practice, leveraging the strengths of each community. Participants will be able to discuss open research problems and identify those of high priority and interest, and join collaborative research teams involving multi-site data collection.

Throughout the seminar breakout groups will investigate diverse topics such as how the needs of professionals and students differ, how metrics for measuring error message performance can be defined, and how to characterize the unique requirements of programmers with disabilities. Attendees are encouraged to collect/organize examples of effective and ineffective error messages, as well as lists/collections of error messages from various languages, including frequency data for given users, and share these prior to the seminar. These will be used during the seminar as a basis for discussion and moving forward. We will include opportunities within the seminar to explore in greater detail any topics identified in early sessions that are of particular interest. This flexibility will enable us to capitalize on the synergies we are confident will develop quickly at this seminar. The wide-ranging issue of problematic error messages will not be solved easily or in short order. This seminar seeks to start this process with a future-focused outlook based on community consensus of the objectives that must be met immediately and a roadmap of concrete steps to take once these are achieved.

Creative Commons BY 4.0

Brett A. Becker, Paul Denny, Janet Siegmund, and Andreas Stefik