Experimental Design for Real-World Systems
Papers from the AAAI Spring Symposium
Katherine Tsui, Chair
Technical Report SS-09-03
76 pp., $30.00
ISBN 978-1-57735-410-9
[Add to Cart] [View Cart]
As more artificial intelligence (AI) research is fielded in real-world applications, the evaluation of systems designed for human-machine interaction becomes critical. AI research often intersects with other areas of study, including human-robot interaction, human-computer interaction, assistive technology, and ethics. Designing experiments to test hypotheses at the intersections of multiple research fields can be incredibly challenging. Many commonalities and differences already exist in experimental design for real-world systems. For example, the fields of human-robot interaction and human-computer interaction are two fields that have both joint and discrete goals. They look to evaluate very different aspects of design, interface, and interaction. In some instances, these two fields can share aspects of experimental design, while, in others, the experimental design must be fundamentally different. This symposium focused on a wide variety of topics that address the challenges of experiment design for real-world systems including successes and failures in system evaluations, uses of quantitative and qualitative data, design of system evaluations, type and size of the participant pool, uses of laboratory experiments, field trials, Wizard of Oz studies, and observational studies, and other related topics.