Benchmarking of Qualitative Spatial and Temporal Reasoning Systems
Papers from the AAAI Spring Symposium
Bernhard Nebel and Stefan Wölfl, Cochairs
Over the past 25 years the domain of qualitative spatial and temporal reasoning has evolved to an established subfield of AI. Qualitative reasoning aims at the development of formalisms that are close to conceptual schematas used by humans for reasoning about their physical environment, in particular, about temporal and spatial information. Application fields of qualitative reasoning include human-machine interaction, high-level agent control, geographic information systems, spatial planning, ontological reasoning, and cognitive modeling.
To foster real-world applications, representation and reasoning methods used in qualitative reasoning need to be tested against evaluation criteria adapted from other AI fields and cognitive science. The aim of the symposium was to boost the development of well-founded and widely accepted evaluation standards and practical benchmark problems. This includes the measures to compare different qualitative formalisms in terms of cognitive adequacy, expressiveness, and computational efficiency; the development of a domain and problem specification language for benchmarking purposes; the identification of significant benchmark domains and problem instances based on natural use cases, as well as the creation of a problem repository; and the measures to evaluate the performance of implemented reasoning systems. The symposium fostered the benchmarking idea in the qualitative reasoning domain, contributed to identify a graded set of challenges for future research, and pushed the development of qualitative reasoning methods and systems towards application-relevant problems.