In computer science as it stands today, there exists thousands of competing programming language products, with new ones (including both domain-specific and general purpose languages) being created regularly. These technologies are important, with applications ranging from educational or business tools to critical infrastructure in society or the military. While programming is by its nature mathematical, meaning that these products have a rigorous foundation to check that they work on a technical level, they are programmed by people that may have challenges and are prone to mistakes. While one might assume that existing communities, like the International Conference on Software Engineering (ICSE), Empirical Software Engineering (ESE), or the programming language conferences themselves (e.g., ICFP, OOPSLA, POPL) might be suitable venues for discussions about human-factors evidence to guide language design, this has not historically been the case. Notably, recent studies by Kaijanaho and others has shown that these venues contain almost no scientific data on the issue, with only 22 studies from the 1950s to 2012. Further, one might reasonably assume that smaller workshops, like the Psychology of Programmers Interest Group (PPIG), older workshops like Empirical Studies on Programmers (ESP), or newer workshops like PLATEAU cover this space, but rigorous investigations into their evidence practices show that these venues rarely include reports with rigorous scientific evidence.

In this Dagstuhl Seminar, we have four primary objectives. First, we are 1) striving to form a diverse community of scholars around programming language design based on evidence of programmer impacts. Given that top-tier venues in the area have not carefully investigated this long-standing and important issue in the field, we want to form a new community that will. Second, given that the design space for programming languages is very large, it makes sense to 2) create an organized set of priorities for new studies in the area along with 3) the evidence standard to be used. Notably, if a new community that contains evidence were to appear, it makes sense to create rigorous standards so that scholars can "test" each other's claims in replication, as opposed to the "trust" model that exists today. Finally, research can often make progress if it adheres to guiding principles, which brings us to our fourth objective, 4) to discuss or create theories that can guide the community in understanding and framing the evidence gathered over time. Historically, scholars have relied on somewhat subjective guiding theories. In contrast, we strive for theories and data that emphasize precision and objectivity.

During the seminar, participants will engage in a variety of activities. First, we will have sessions on the existing literature to provide attendees with historical context, a shared sense of what evidence exists, and methods that have been used to gather that evidence. Second, as there are considerable holes in the scholarly literature, we will generate lists of possible studies and try to prioritize them as a group. Third, to better understand what evidence standard could be used for these future studies, we will discuss evidence standards as they exist in other scholarly disciplines and their respective pros and cons. In summary, seminar participants will investigate the impact of programming language design on the productivity of programmers using systematic science with established standards.

Creative Commons BY 3.0 DE

Stefan Hanenberg, Brad A. Myers, Bonita Sharif, and Andreas Stefik

• Jonathan Aldrich (Carnegie Mellon University - Pittsburgh, US) [dblp]
• Craig Anslow (Victoria University - Wellington, NZ) [dblp]
• Ameer Armaly (University of Notre Dame, US) [dblp]
• Johannes Bechberger (KIT - Karlsruher Institut für Technologie, DE)
• Brett A. Becker (University College Dublin, IE) [dblp]
• Andrew Begel (Microsoft Research - Redmond, US) [dblp]
• Tanja Blascheck (INRIA Saclay - Orsay, FR) [dblp]
• Neil Brown (King's College London, GB) [dblp]
• Michael Coblenz (Carnegie Mellon University - Pittsburgh, US) [dblp]
• Igor Crk (Southern Illinois Univ. - Edwardsville, US) [dblp]
• John M. Daughtry (Google - Seattle, US) [dblp]
• Fabian Deitelhoff (Fachhochschule Dortmund, DE) [dblp]
• Rob DeLine (Microsoft Corporation - Redmond, US) [dblp]
• Brian Dorn (University of Nebraska - Omaha, US) [dblp]
• Andrew Duchowski (Clemson University, US) [dblp]
• Scott Fleming (University of Memphis, US) [dblp]
• Baker Franke (Code.org - Seattle, US) [dblp]
• Reiner Hähnle (TU Darmstadt, DE) [dblp]
• Matthias Hauswirth (University of Lugano, CH) [dblp]
• Felienne Hermans (TU Delft, NL) [dblp]
• Johannes C. Hofmeister (Universität Passau, DE) [dblp]
• Ciera Jaspan (Google Inc. - Mountain View, US) [dblp]
• Antti-Juhani Kaijanaho (University of Jyväskylä, FI) [dblp]
• A. J. Ko (University of Washington - Seattle, US) [dblp]
• Thomas D. LaToza (George Mason University - Fairfax, US) [dblp]
• Andrew Macvean (Google - Seattle, US) [dblp]
• Jonathan I. Maletic (Kent State University, US) [dblp]
• Amelia A. McNamara (Smith College - Northampton, US) [dblp]
• Briana B. Morrison (University of Nebraska - Omaha, US) [dblp]
• Brad Myers (Carnegie Mellon University - Pittsburgh, US) [dblp]
• Lutz Prechelt (FU Berlin, DE) [dblp]
• Sibylle Schupp (TU Hamburg-Harburg, DE) [dblp]
• Bonita Sharif (Youngstown State University, US) [dblp]
• Andreas Stefik (Univ. of Nevada - Las Vegas, US) [dblp]
• Walter F. Tichy (KIT - Karlsruher Institut für Technologie, DE) [dblp]
• Phillip Merlin Uesbeck (Univ. of Nevada - Las Vegas, US) [dblp]
• Lea Verou (MIT - Cambridge, US) [dblp]