TOP
Search the Dagstuhl Website
Looking for information on the websites of the individual seminars? - Then please:
Not found what you are looking for? - Some of our services have separate websites, each with its own search option. Please check the following list:
Schloss Dagstuhl - LZI - Logo
Schloss Dagstuhl Services
Seminars
Within this website:
External resources:
  • DOOR (for registering your stay at Dagstuhl)
  • DOSA (for proposing future Dagstuhl Seminars or Dagstuhl Perspectives Workshops)
Publishing
Within this website:
External resources:
dblp
Within this website:
External resources:
  • the dblp Computer Science Bibliography


Please select your role
to obtain appropriate quick-links.
As a LIPIcs/OASIcs author, you will be invited by e-mail to register at the Dagstuhl Submission Server. The server will guide you through the publication workflow. Preliminary information can be found here:
Quick Links

Dagstuhl Perspectives Workshop 16252

Engineering Academic Software

( Jun 19 – Jun 24, 2016 )
(Click in the middle of the image to enlarge)
Permalink
Please use the following short url to reference this page: https://www.dagstuhl.de/16252
Organizers
Coordinator
Contact
Publications
Impacts
Motivation
Show Motivation

Software is fundamental to academic research work, both as part of the method and as the result of research. With the advent of artifact evaluation committees of conferences, journals which include source code and running systems as part of the published artifacts, as well as the increasing push to reproducibility, we foresee that software will only increase in importance as part of the academic process. To ensure the quality of this software, we must

  1. better understand the constraints of the environment in which academic software is created and used,
  2. adapt and distribute appropriate software engineering knowledge and expertise across communities, and
  3. identify strengths, weaknesses, risks and opportunities in academic software engineering.

The goal of this Perspectives Workshop is to produce a roadmap towards future professional software engineering for software-based research instruments and other software produced and used in an academic context (i.e., for science, not administration). We will seek to address the following questions:

  • What is academic software? How is it different from other software? What are its most pressing dimensions of quality?
  • Is the software that drives our research methods correct? Are the inputs and outputs sufficiently specified to be able to interpret the difference between incorrect and correct? How to verify (test or prove) our claims?
  • Is the software we use and produce in an academic context sustainable? How can we ensure that the software continues to evolve and offer value after serving its initial purpose?
  • Are we able to incrementally adapt research software to emerging opportunities at the same time, without loss of reproducibility and without incurring prohibitive cost?
  • How can we adapt software engineering methods for the unique academic context without losing quality?
  • How can we balance domain knowledge and expertise with software engineering knowledge and expertise in an academic research team?
  • What are motivators for investment and change for research teams, considering the highly competitive and already complex environment they operate in? Is it clear what is required in terms of long term funding, education and infrastructure to make the goals of academic software feasible?
  • Do quality aspects of academic software apply to open data as well? How can well-engineered academic software together with open data make science more reproducible?

We conjecture software engineering and academic software developers can learn from each other. We believe that the software engineering research community is well-positioned to provide input on research software for other research communities. At the same time, the software engineering community will learn from other research communities who have experience in validating experimental research methods, calibrating research instruments, and developing software for and with academic researchers.

Press room
Show Press room
Summary
Show Summary

This Dagstuhl Perspectives Workshop brought together activists, experts and stakeholders on the subject of high quality software produced in an academic context. Our current dependence on software across the sciences is already significant, yet there are still more opportunities to be explored and risks to be overcome. The academic context is unique in terms of its personnel, its goals of exploring the unknown and its demands on quality assurance and reproducibility.

We refer to the IEEE Internet Computing article "Better Software, Better Research" [1] which motivated the topic. In this workshop we took the following perspective of a research team which is in either or both of the following situations:

  • consuming or producing software as an output of the academic process;
  • consuming or producing software as emph{a component} of the research methods.

Society is now in the tricky situation where several deeply established academic fields (eg physics, biology, mathematics) are shifting towards dependence on software, programming technology and software engineering methodology which are backed only by young and rapidly evolving fields of research (computer science and software engineering). Full accountability and even validity of software-based research results are now duly being challenged.

With the outputs of this interactive and productive perspectives workshop, we strive to contribute in a positive manner to the above challenges. We formulated taxonomies with definitions to clarify the domain, we co-authored concrete policy and process documents to improve the status and recognition of academic software development and academic software engineers, and finally we formulated a list of 18 concrete declarations of intent ("I wil" pledges). This list was presented to the WSSSPE community [2] in September 2016 to acquire feedback and it will be the backbone of the Dagstuhl Manifesto document we are editing. It serves to motivate change by proposing policy changes with concrete actions and instilling positive attitudes towards academic software.

Participants.The participants of the workshop came from three major groups. The first group consists of active and visible members of the global academic software engineering community. They represent (formal) institutions such as the Software Sustainability Institute, the Software Carpentry Foundation, and eScience and data science centers from across the globe. The second group contributed researchers in empirical software engineering, with a specific eye on studying the principles and practices of academic software engineering. The final group contributed researchers as an audience: software engineering researchers with a long experience in engineering software for software itself or software for specific academic research fields.

We found that without exception the participants were strongly motivated and able to actively contribute to the proceedings of the workshop; the mix of people proved to be well-balanced. This balance is an accomplishment, given that invitees from computer science were far more likely to know of Dagstuhl workshops than other groups. To attest to our outcomes we've selectively listed three (paraphrased) verbal statements here:

  • "The workshop was a transformational experience for me; I've learned an entire new perspective on my field and I intend to apply the insights in my daily practice."
  • "I had an epiphany yesterday after dinner; now I understand how to connect the data science research at my university to the computer science department."
  • "Before the workshop I had no idea so many initiatives were already underway in [improving] academic software engineering; this has given my understanding of the challenges a real boost and I know what the some of the next steps to take are."

Schedule. The schedule of the workshop was designed to maximize both interactive discussion and work towards tangible outputs. Key points were: to start the day with inspiring presentations to set the stage, then to have at least 40\% of the day time allocated to free discussion time, and to explicitly share successes (output) of each day's breakout groups in a plenary session.

The workshop started on Monday with a quick and tightly timed round of 2 minute personal introductions. Otherwise on Monday, Tuesday and Thursday the program was structured equally: in the morning we would have plenary presentations which included exploratory discussions. These sessions were meant to bring everybody up-to-speed with ongoing and past initiatives. During and after lunch we used a board with sticky notes to define break-out groups. Each break-out group was centred around a specific discussion topic and (usually) a specific idea for an output document was associated with it. After coffee we would go back to the same break-out group to collaboratively record the notes and lessons from each group (stored in a shared online document). Between 17:00 and 18:00 we reconvened and harvested the results of each breakout group with the others. People could and did freely switch between breakout groups but this was not a common thing.

On Wednesday we had an "open-mic" session with 8 presentations of around 10 minutes, sharing experiences and results, before we had a long walk in the surroundings. The organizers also designed an initial skeleton structure and ideas for the manifesto that day.

On Thursday afternoon and Friday morning we all worked together on our Dagstuhl Manifesto by first reworking our notes into the ideas around the manifesto, specifically a list of "I will" pledges with references and motivation. Finally, Friday afternoon a small remaining group re-ordered the group's manifesto notes into a well-structured list of 18 pledges. Two of the organizers remained to continue to edit the current report and the manifesto document.

Output. Output documents of the workshop are organized under the "DagstuhlEAS" organisation on GitHub. This currently features 6 draft documents, including the current report and (a) the manifesto, (b) the Research Software Engineering Handbook, (c) a Literature Survey, (d) a Taxonomy on Software Credit Roles, and (e) a Software Award Proposal. Next to these documents, an R&D project proposal was produced on measuring the impact of academic software.

The remainder of this document summarizes the morning sessions by listing the abstracts of each talk, the afternoon breakouts by describing each topic and its results, and finally the research questions on the topic of engineering academic software we have collected.

References

  1. Carole Goble. Better software, better research. IEEE Internet Computing, 18(5):4–8, Sep 2014.
  2. Alice Allen, Cecilia Aragon, Christophe Becker, Jeffrey C. Carver, Andrei Chis, Benoit Combemale, Mike Croucher, Kevin Crowston, Daniel Garijo, Ashish Gehani, Carole Goble, Robert Haines, Robert Hirschfeld, James Howison, Kathryn Huff, Caroline Jay, Daniel S. Katz, Claude Kirchner, Kateryna Kuksenok, Ralf Lämmel, Oscar Nierstrasz, Matthew Turk, Rob van Nieuwpoort, Matthew Vaughn, and Jurgen Vinju. Lightning talk: “I solemnly pledge” – a manifesto for personal responsibility in the engineering of academic software. In Proceedings of the Fourth Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE4).
Copyright Carole Goble and James Howison and Claude Kirchner and Oscar M. Nierstrasz and Jurgen J. Vinju
Participants
Show Participants
  • Alice Allen (University of Maryland - College Park, US) [dblp]
  • Cecilia Aragon (University of Washington - Seattle, US) [dblp]
  • Christoph Becker (University of Toronto, CA) [dblp]
  • Jeffrey Carver (University of Alabama, US) [dblp]
  • Andrei Chis (Universität Bern, CH) [dblp]
  • Benoit Combemale (IRISA - Rennes, FR) [dblp]
  • Mike Croucher (Software Sustainability Institute - Edinburgh, GB)
  • Kevin Crowston (Syracuse University, US) [dblp]
  • Daniel Garijo (Technical University of Madrid, ES) [dblp]
  • Ashish Gehani (SRI - Menlo Park, US) [dblp]
  • Carole Goble (University of Manchester, GB) [dblp]
  • Robert Haines (University of Manchester, GB) [dblp]
  • Robert Hirschfeld (Hasso-Plattner-Institut - Potsdam, DE) [dblp]
  • James Howison (University of Texas - Austin, US) [dblp]
  • Katy Huff (University of California - Berkeley, US) [dblp]
  • Caroline Jay (University of Manchester, GB) [dblp]
  • Daniel S. Katz (University of Illinois at Urbana Champaign, US) [dblp]
  • Claude Kirchner (INRIA - Le Chesnay, FR) [dblp]
  • Katie Kuksenok (University of Washington - Seattle, US) [dblp]
  • Ralf Lämmel (Universität Koblenz-Landau, DE) [dblp]
  • Oscar M. Nierstrasz (Universität Bern, CH) [dblp]
  • Matthew J. Turk (University of Illinois at Urbana Champaign, US) [dblp]
  • Rob van Nieuwpoort (VU University Amsterdam, NL) [dblp]
  • Matthew Vaughn (University of Texas - Austin, US) [dblp]
  • Jurgen J. Vinju (CWI - Amsterdam, NL) [dblp]
Related Seminars
  • Dagstuhl Seminar 24161: Research Software Engineering: Bridging Knowledge Gaps (2024-04-14 - 2024-04-19) (Details)
Classification
  • bioinformatics
  • data bases / information retrieval
  • software engineering
Keywords
  • Software
  • Software Eng
  • Research Methods
  • Sustainable Software
Seminar 16252
© 2024 Schloss Dagstuhl – Leibniz-Zentrum für Informatik GmbH
ImprintContactAccessibilityPrivacy