In many subfields of computer science (CS), experiments play an important role. Besides theoretic properties of algorithms or methods, their effectiveness and performance often can only be validated via experimentation. In most of these cases, the experimental results depend on the input data, settings for input parameters, and potentially on characteristics of the computational environment where the experiments were designed and run. Unfortunately, most computational experiments are specified only informally in papers, where experimental results are briefly described in figure captions; the code that produced the results is seldom available.

This has serious implications. Scientific discoveries do not happen in isolation. Important advances are often the result of sequences of smaller, less significant steps. In the absence of results that are fully documented, reproducible, and generalizable, it becomes hard to re-use and extend these results. Besides hindering the ability of others to leverage our work, and consequently limiting the impact of our field, the absence of reproducibility experiments also puts our reputation at stake, since reliability and validity of empiric results are basic scientific principles.

Reproducible results are not just beneficial to others – in fact, they bring many direct benefits to the researchers themselves. Making an experiment reproducible forces the researcher to document execution pathways. This in turn enables the pathways to be analyzed (and audited) and it also helps newcomers (e.g., new students and post-docs) to get acquainted with the problem and tools used. Reproducibility also forces portability which simplifies the dissemination of the results. Last, but not least, preliminary evidence exists that reproducibility increases impact, visibility and research quality.

However, attaining reproducibility for computational experiments is challenging. It is hard both for authors to derive a compendium that encapsulates all the components (e.g., data, code, parameter settings, environment) needed to reproduce a result, and for reviewers to verify the results. There are also other barriers, from practical issues – including the use of proprietary data, software and specialized hardware, to social – for example, the lack of incentive for authors to spend the extra time making their experiments reproducible.

In this seminar, we will bring together experts from various sub-fields of Computer Science (CS) to create a joint understanding of the problems of reproducibility of experiments, discuss existing solutions and impediments, and propose ways to overcome current limitations. Some topics we intend to cover include, but are not limited to: reproducibility requirements, infrastructure, new CS research directions required to support reproducibility, incentives, education. Each participant is expected to present the state of the art, requirements, and issues related to reproducibility in their sub-field. Workgroups will be formed to discuss cross-cutting issues.

The expected outcome of the seminar will be a manifesto proposing guidelines, procedures, and further activities for improving reproducibility and broadening its adoption in CS.

More specifically, we will address the following key issues:

• Understanding and requirements
• Technical aspects of repeatability
• Benchmarks and worksets
• IPR, public availability of research, non-consumptive research
• Infrastructures
• New challenges and research directions
• Specialization and integration across disciplines
• Awareness, education and communication
• Incentives for repeatability

• Vanessa Braganholo (Fluminense Federal University, BR) [dblp]
• Fernando Chirigati (NYU Tandon School of Engineering, US) [dblp]
• Christian Collberg (University of Arizona - Tucson, US) [dblp]
• Shane Culpepper (RMIT University - Melbourne, AU) [dblp]
• David De Roure (University of Oxford, GB) [dblp]
• Arjen P. de Vries (Radboud University Nijmegen, NL) [dblp]
• Jens Dittrich (Universität des Saarlandes, DE) [dblp]
• Nicola Ferro (University of Padova, IT) [dblp]
• Juliana Freire (New York University, US) [dblp]
• Norbert Fuhr (Universität Duisburg-Essen, DE) [dblp]
• Daniel Garijo (Technical University of Madrid, ES) [dblp]
• Carole Goble (University of Manchester, GB) [dblp]
• Kalervo Järvelin (University of Tampere, FI) [dblp]
• Noriko Kando (National Institute of Informatics - Tokyo, JP) [dblp]
• Randall J. LeVeque (University of Washington - Seattle, US) [dblp]
• Matthias Lippold (Universität Duisburg-Essen, DE) [dblp]
• Bertram Ludäscher (University of Illinois at Urbana-Champaign, US) [dblp]
• Mihai Lupu (TU Wien, AT) [dblp]
• Tanu Malik (University of Chicago, US) [dblp]
• Rudolf Mayer (SBA Research - Wien, AT) [dblp]
• Alistair Moffat (The University of Melbourne, AU) [dblp]
• Kevin Page (University of Oxford, GB) [dblp]
• Raul Antonio Palma de Leon (Poznan Supercomputing and Networking Center, PL)
• Martin Potthast (Bauhaus-Universität Weimar, DE) [dblp]
• Andreas Rauber (TU Wien, AT) [dblp]
• Paul Rosenthal (TU Chemnitz, DE) [dblp]
• Claudio T. Silva (New York University, US) [dblp]
• Stian Soiland-Reyes (University of Manchester, GB) [dblp]
• Benno Stein (Bauhaus-Universität Weimar, DE) [dblp]
• Rainer Stotzka (KIT - Karlsruher Institut für Technologie, DE) [dblp]
• Evelyne Viegas (Microsoft Research - Redmond, US) [dblp]
• Stefan Winkler-Nees (DFG - Bonn, DE)
• Torsten Zesch (Universität Duisburg-Essen, DE) [dblp]
• Justin Zobel (The University of Melbourne, AU) [dblp]