For many applications of smart embedded software systems, the system should sense the footprint of a human or humans acting in the system's environment, interpret the sensor data in terms of some semantic model about what the human is doing, and respond appropriately in real time. Examples of such applications include smart homes, human-machine or human-robot interaction, assistance, surveillance, and tutoring systems; given the current trend towards ambient intelligence, ubiquitous computing, and sensor networks, the number of systems in these categories can certainly by expected to rise in the next ten years or so.

The problem shares many features with classical object recognition and scene reconstruction from sensor data in terms of a static scene model. Interpreting in semantic terms sensor data from the environment has a long tradition in AI - arguably, it was one of the original core problems put forth by AI's founding fathers. However, the problem of interpreting observed action in the sense of the seminar differs in some aspects from what state-of-the-art AI or engineering approaches would allow to be tackled by routine:

• Events in space-time rather than static objects need to be characterized. This necessarily involves some representation and model of temporal and spatial data (e.g., the human put a saucepan on the cooker and then turned the cooker on).
• Real-time processing of the sensor data or percepts is required to keep track of what is happening. In fact, "real time" here is the pace of human action, i.e., relatively slow compared to CPU clock ticks. However, given a potentially rich stream of sensor data and a potentially large body of background knowledge, even this pace is demanding for the respective knowledge processing methods.
• Willed human action, be it planned, intended, or customary, is the domain of interpretation. In Knowledge Representation, this is a relatively unexplored area, compared to, say, upper ontologies of household items, red wine, or pizza varieties.

Contemplating the three words that make up the title of this seminar ("interpret", "observe", and "act"), it becomes clear that there are a number of issues that need to be addressed in this context. Firstly, any interpretation is to some degree subjective and uses a particular repertoire of basic actions in its language. Secondly, an observation uses a particular type of sensor data and often is not possible without interpretation at the same time. Thirdly, there are issues around what actions are to be considered:

• Are only willed and physical action to be considered?
• Is avoidance an action?
• What constitutes an action in the first place?
• When does a particular action end?
• Is an unsuccessful action an action?

In summary, what precisely is observed action interpretation and what would be benchmark data for it?

To find an answer to this question, the participants of the seminar emerged themselves in a variety of activities: technical talks, working groups, plenary discussions, and a number of informal discussions. In the rest of this report, some of these activities and their results are discussed in more detail.