Development of multimodal applications is an iterative, complex, and often a rather heuristic process. This is because in multimodal systems the number of interplaying components can be far greater than in an unimodal Spoken Dialogue System. From the developer's perspective, a multimodal system presents challenges and technical difficulties on many levels. In this paper we will describe our approach to one specific component of multimodal systems, the Multimodal Integrator. On the other hand, from the designer's perspective, all components must be fine-tuned to a level that their combined overall performance can deliver the desired experience to end users. In both cases, evaluation and analysis of the current implementation is paramount. Hence, looking into the details while getting a good understanding of the overall performance of a multimodal system is the other key topic.

We describe TickTock, a conversational agent designed to engage humans on topics of its choosing and to carry on an interaction for as long as possible. Our prototype uses a database of talk show transcripts featuring guests from the film industry. To be an interesting companion Tick Tock uses immediate context from the last two turns to formulate queries into a database of utterances. The process is automatic. TickTock monitors user engagement and performs certain moves, such as topic shifts, based on its assessment of user state. Initially we used utterance content for monitoring and subsequently we begun to investigate non-language cues, such as prosody and visual cues to create a more robust engagement model based on multiple human communication channels.

Several recent approaches use distributional similarity for making symbolic reasoning more flexible. While an important step in the right direction, the use of similarity has a number of inherent limitations. We argue that similarity-based reasoning should be complemented with commonsense reasoning patterns such as interpolation and a fortiori inference. We show how the required background knowledge for these inference patterns can be obtained from distributional models.

We present an emerging research project in our laboratory to extend ambient intelligence (AmI) by what we refer to as “extreme personalization” meaning that an instance of ambient intelligence is focused on one or at most a few individuals over a very long period of time. Over a lifetime of co-activity, it senses and adapts to a person’s preferences and experiences, and crucially, his or her (changing) special needs; needs that differ significantly from the normal baseline. We refer to our agent-based cyber-physical system as Ambient Personal Environment eXperiment (APEX). It aims to serve as a Companion , a Coach , and a Caregiver : crucial support for individuals with mental, physical, and age-related disabilities and those other people who help them. We propose that an instance of APEX, interacting socially with each of these people, is both a social actor as well as a cyber-human prosthetic device . APEX is an ambitious integration of multiple technologies from Artificial Intelligence (AI) and other disciplines. Its successful development can be viewed as a grand challenge for AI. We discuss in this paper three research thrusts that lead toward our vision:  robust intelligent agents, semantically rich human-machine interaction, and reasoning from comprehensive multi-modal behavior data.

In this paper we explore the use of visual common-sense knowledge and other kinds of knowledge (such as domain knowledge, background knowledge, linguistic knowledge) for scene understanding. In particular, we combine visual processing with techniques from natural language understanding (especially semantic parsing), common-sense reasoning and knowledge representation and reasoning to improve visual perception to reason about finer aspects of activities.

This paper first sketches a living lab infrastructure installed in an alternative housing unit built to host 10 people with traumatic brain injury. It then presents the first research project in progress within this living lab. This interdisciplinary project aims at designing, implementing, deploying, and assessing a personalized assistive technology (PAT). Based on the needs and expectations expressed by the residents, their caregivers and their families, a cooking assistant appeared as one of the best suited PAT to foster residents autonomy and social participation. The resulting PAT will rely on pervasive computing and ambient intelligence. It will then be personalized according to each participant's capacities and specific cognitive impairments. The impact of the assistant on autonomy and quality of life will then be measured. The overall organizational impact of such assistive technology will be also documented and evaluated.

We explore how children with autism form teams and what kind of difficulties they experience. Autistic reasoning is an adequate means to explore team formation because it is rather simple compared to the reasoning of controls and software systems on one hand, and allows exploration of human behavior in real-world environment on the other hand. We discover that reasoning about mental world, impaired in various degrees in autistic patients, is the key parameter of limiting the capability to form teams and cooperate. While teams of humans, robots and software agents have a manifold of limitations to form teams, including resources, conflicting desires, uncertainty, environment constraints, children with autism have only single limitation which is reduced reasoning about mental world. We correlate the complexity of the expressions for mental states children are capable of operating with their ability to form teams. Reasoning rehabilitation methodology is described, as well as its implications for children behavior in real world including cooperation and team formation.

In this abstract, we first consider abduction in human dialogues. Two different types of abduction, objective abduction and subjective abduction, are introduced and formulated using propositional modal logic. We next formulate conversational implicature in the same logic and contrast it with abduction in dialogues. According to our formulation, abduction uses private belief of a reasoner, while conversational implicature relies on common knowledge between participants in conversation. The results characterize how hearers use abduction or conversational implicatures to figure out what speakers have implicated and show how two commonsense inferences are distinguished.

This paper describes a modular action language, ALM, dedicated to the specification of complex dynamic systems. One of the main goals of the language is to facilitate the development and testing of knowledge representation libraries. We present the implementation of a large scale library of commonsense concepts, achieved by porting knowledge from the Component Library (CLIB) into ALM. Our choice of CLIB as a source of inspiration is justified by the well-founded methodology used by its authors in selecting the general concepts it contains, and its extensive testing in the context of the Automated User-centered Reasoning and Acquisition System. The resulting ALM library has the additional advantage of incorporating established knowledge representation methodologies developed in the action language research community.

The crisp/brittle semantic model behind databases limits the scale in which data consumers can query, explore, integrate and process structured data. Approaches aiming to provide more comprehensive semantic models for databases, which are purely logic-based (e.g. as in Semantic Web databases) have major scalability limitations in the acquisition of structured semantic and commonsense data. This work describes a complementary semantic model for databases which has semantic approximation at its center. This model uses distributional semantic models (DSMs) to extend structured data semantics. DSMs support the automatic construction of semantic and commonsense models from large-scale unstructured text and provides a simple model to analyze similarities in the structured data. The combination of distributional and structured data semantics provides a simple and promising solution to address the challenges associated with the interaction and processing of structured data.