Hierarchical Multimodal Planning for Pervasive Interaction

Traditional dialogue management systems are tightly coupled with the sensing ability of a single computer. How to organize an interaction in pervasive environments to provide a friendly and integrated interface to users is an important issue. This requires a transition of the human-computer interaction (HCI) from tight coupling to loose coupling. This paper proposes a hierarchical multimodal framework for pervasive interactions. Our system is designed to remind the activities of daily living for individuals with cognitive impairments.The system is composed of Markov decision processes for activity planing, and multimodal partially observable Markov decision processes for action planning and executing. Empirical results demonstrate the hierarchical multimodal framework establishes a flexible mechanism for pervasive interaction systems.