Estimating User's Attitude in Multimodal Conversational System for Elderly People with Dementia

Toward constructing a multimodal conversation agentsystem which can be used to interview elderly patients with dementia, we propose a turn taking mechanism based on recognition of the subjects attitude as to whether the subject has (or relinquish) the right to speak. A key strategy in the recognition task is to extract features from pausing behavior in subject’s spontaneous speech and to fuse multimodal signals (gaze, head motion, and speech). In this paper, we focus on evaluation of the recognition module used in guiding turn taking. To evaluate it, we collect multimodal data corpus from 42 dyadic conversations between subjects with dementia and the virtual agent we have developed as a prototype and annotate subject’s multimodal data manually. In experiments, we validate recognition models trained multimodal dataset by machine learning methods.Experimental results shows that pause features are effective to improve the attitude recognition accuracy and the accuracy is improved up to 88%.