Performance Evaluation is a discipline of Computer Science for some thirty years. It seems time to take stock of what we were doing. That is, provide answers to the following questions:

• What are its scientific contributions?
• What is its relevance in industry and business?
• What is its standing in academia?
• Where is the field headed?
• What are its success stories and failures?
• What are its current burning questions?

At this Dagstuhl seminar, we were taking the term workshop literally and have broken up in five working groups. These groups worked in parallel on different areas of Performance Evaluation. Each group prepared a written report during the stay at Dagstuhl. These reports constitute the remainder of this text. The ultimate goal of this seminar is to publish a White Book on Performance Evaluation that will become a catalyst for the advancements of this field as well as a book of reference. Furthermore, each group discussed Success Stories, Failures, Impact of Methods, and Burning Questions of Performance Evaluation. A summary of this discussion is given in the following:

Success Stories

• Computer architecture, in particular RISC, cache coherence, and multiprogramming.
• IEEE 802 suite of protocols, in particular medium access control level.
• Data networks, in particular TCP/IP.
• Storage systems, in particular RAID and virtual memory.
• Operating systems, in particular timesharing, MVS, and scheduling.
• Realization and implementation of methodological results in easy-to-use software packages.

Failures

• Late consideration of performance issues in system design process.
• Bad quality of measured input data.
• Wrong level of abstraction.
• Educational failures
  • Lack of making known methods to customers.
  • Lack of training of modeling skills

Impact of Methods

• BCMP theorem made impact to e.g., capacity planning, multiprogramming, and communication networks.
• MVA algorithm substantially increased the applicability of BCMP queueing networks with several customer classes for supporting the design of computer and communication systems.
• Particular stochastic models with closed-form solutions made impact to design of IEEE 802 medium access control protocols.
• Trace-driven simulation made impact to cache and processor architecture design

Burning Questions

• Complexity of today’s systems
  • Scalability of performance evaluation techniques.
  • Validation
  • Parameter estimation, large amount of measured data.
  • Develop methods for handling very large models with sufficient level of detail
• Bridging the gap between theoretical and applied work:
  • Seamless integration of performance models into design process.
  • Application-specific parametrizable performance models.
• Academic education and university research:
  • Curriculum for computer scientists and engineers has not enough exposure to Performance Evaluation methods.
  • Faculty positions and research funding is inadequate for Performance Evaluation methodology. I.e., is pure Performance Evaluation research an endangered species?
• Acceptance in industry:
  • Performance Evaluation experts must be routinely be involved in system design process.
  • Short development cycle, short live cycles of technologies.
  • Education of university graduates.