We discuss experiences from evaluating the learning performance of a user-adaptive personal assistant agent.  We discuss the challenge of designing adequate evaluation and the tension of collecting adequate data without a fully functional, deployed system.  Reflections on negative and positive experiences point to the challenges of evaluating user-adaptive AI systems.  Lessons learned concern early consideration of evaluation and deployment, characteristics of AI technology and domains that make controlled evaluations appropriate or not, holistic experimental design, implications of "in the wild" evaluation, and the effect of AI-enabled functionality and its impact upon existing tools and work practices.

We describe an AI-enhanced task management tool developed for a military environment, which differs from office environments in important ways: differing time scales, a focus on teams collaborating on tasks instead of an individual managing her own set of diverse tasks, and a focus on tasklists and standard operating procedures instead of individual tasks. We discuss the Task Assistant prototype, our process for adapting it from an office environment to a military one, and lessons learned about developing AI technology for a high-pressure operational environment.

Interactive artificial intelligence systems employ preferences in both their reasoning and their interaction with the user. This survey considers preference handling in applications such as recommender systems, personal assistant agents, and personalized user interfaces. We survey the major questions and approaches, present illustrative examples, and give an outlook on potential benefits and challenges.

Interactive artificial intelligence systems employ preferences in both their reasoning and their interaction with the user. This survey considers preference handling in applications such as recommender systems, personal assistant agents, and personalized user interfaces. We survey the major questions and approaches, present illustrative examples, and give an outlook on potential benefits and challenges.