Experimental research on networked systems requires a suitable research infrastructure to perform experiments. For many years, the dominating approach of research groups was to create a specific experimental setup tailored to the needs of specific experiments, e.g., in the context of a PhD thesis. The community is well aware of the obvious shortcomings of this approach. One shortcoming is that while the time and effort to set up the needed infrastructure is high, the fact that the setups are created independently not only means unnecessary duplication of work, but also heterogeneity, with details frequently not documented in publications, that may lead to difficulties reproducing experiments of other scientists.

The networked systems community built large testbed research infrastructures, which contribute to overcome such challenges, and which provide valuable resources for experimental research of networked systems. However, there appears to be a gap between what the community that operates research infrastructures provides as services, and what scientists performing cuttingedge experimental research actually need. One area of possible improvement is experiment control. So far, scientists who use a large testbed for specific experiments need to solve how to orchestrate their experiments, how to collect, process, and store the data produced by the experiment, and how to add metadata to support other scientists. Consequently, there is a high heterogeneity with respect to the artifacts of specific experiments. In the typical workflow that is required when a paper with artifacts (software and data with metadata) is to be evaluated by an artifact evaluation committee, this heterogeneity and lack of generally available tools for the experiment workflow lead to a high effort for those who prepare the artifacts, for those who review the artifacts (as part of the evaluation committee work), and also for those who want to make use of the artifacts, e.g. when comparing own artifacts with previously published artifacts.

This Dagstuhl Seminar aims to find ways to increase the value of these research infrastructures, by identifying improved ways for collaboration by using testbeds and software tools for experiment workflows, aiming for experiments and their evaluation software to become easy to modify, extend, combine, and port to other testbed environments. By identifying improved frameworks for collaboration, with tools in support of experiment workflows, the seminar aims to support the research community in increasing quality, reproducibility and reuse of research results, and to identify innovative toolchains that increase the productivity of scientists and allow for a more efficient use of research infrastructure resources.

In the seminar, we want to identify approaches that combine a set of measures suitable to overcome the identified problems, by addressing all phases of experiment design, execution, data and metadata generation, processing, evaluation, and publication.

Creative Commons BY 4.0

Georg Carle, Serge Fdida, Kate Keahey, and Henning Schulzrinne