In the black-box optimization community, benchmarking is the established method for empirical algorithm analysis and systematic comparison. Most of the time the community uses the well-studied "classical BBOB" suite with 24 scalable test problems. These were originally proposed in 2009 by Hansen et. al. and chosen carefully to tease out certain algorithmic properties and facilitate principled algorithm comparisons. In addition to the suite of functions, the BBOB group also proposed and published an experimental methodology, largely encoded in the COCO framework.

Currently, a paradigm shift is taking place from human interpretation of results towards machine interpretation. The long-term vision here is to automate the benchmarking process further to enable machines to automatically learn the properties of algorithms. This requires a more rigid benchmarking process with all assumptions and design decisions made explicit and accessible for machine interpretation.

In this workshop, we therefore want to reevaluate some of the assumptions made 15 years ago and make the design decisions and intentions that went into picking the experimental methodology and function set more transparent. The goal is to develop guidelines for stating benchmarking intent, specifying algorithm configuration and documenting benchmark execution. We want to eliminate as much guesswork as possible from the benchmarking and analysis process by making all design decisions explicit. As an example, we will study the evolution of the BBOB suite into the SBOX-COST suite which explicitly specified bounds/box constraints and adjusted the distribution of the optima in the search space.

Creative Commons BY 4.0

Anna Kononova and Olaf Mersmann