Search |  Contact |  SRI Home Do not follow this link, or your host will be blocked from this site. This is a spider trap. Do not follow this link, or your host will be blocked from this site. This is a spider trap. Do not follow this link, or your host will be blocked from this site. This is a spider trap.A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A ASRI International.  333 Ravenswood Avenue.  Menlo Park, CA 94025-3493. SRI International is a nonprofit corporation.

Publication Details

Evaluating User-Adaptive Systems: Lessons from Experiences with a Personalized Meeting Scheduling Assistant

by Berry, P. and Donneau-Golencer, T. and Duong, K. and Gervasio, M. and Peintner, B. and Yorke-Smith, N.

in Proceedings of the Twenty-First International Conference on Innovative Applications of Artificial Intelligence (IAAI’09)

Published by AAAI Press
Address: Pasadena, CA
July 2009.

Abstract

We discuss experiences from evaluating the learning performance of a user-adaptive personal assistant agent. We discuss the challenge of designing adequate evaluation and the tension of collecting adequate data without a fully functional, deployed system. Reflections on negative and positive experiences point to the challenges of evaluating user-adaptive AI systems. Lessons learned concern early consideration of evaluation and deployment, characteristics of AI technology and domains that make controlled evaluations appropriate or not, holistic experimental design, implications of "in the wild" evaluation, and the effect of AI-enabled functionality and its impact upon existing tools and work practices.

Electronic Copies


Adobe PDF

BibTeX

EndNote
Copyright ©2009 AAAI

Associated Projects

CALO

Cognitive Assistant that Learns and Organizes

As part of DARPA’s Personalized Assistant that Learns (PAL) program, SRI and team members are working on developing a next-generation "Cognitive Agent that Learns and Organizes" (CALO).

AIC Personnel

Name Title E-mail
Berry, Pauline M Alumnus
Donneau-Golencer, Thierry David Alumnus
Duong, Khang Software Engineer II
Gervasio, Melinda Senior Computer Scientist
Peintner, Bart Sr. Computer Scientist
Yorke-Smith, Neil Computer Scientist

SRI International
©2014 SRI International 333 Ravenswood Avenue, Menlo Park, CA 94025-3493
SRI International is an independent, nonprofit corporation. Privacy policy