Abstract-- Incorporating empathetic behavior into robots can improve their social effectiveness and interaction quality. In this paper, we present whEE (when and how to express empathy), a framework that enables social robots to detect when empathy is needed and generate appropriate responses. Using large language models, whEE identifies key behavioral empathy cues in human interactions. We evaluate it in human-robot interaction scenarios with our social robot, Haru. Results show that whEE effectively identifies and responds to empathy cues, providing valuable insights for designing social robots capable of adaptively modulating their empathy levels across various interaction contexts. In most scenarios, Large Language Models (LLMs) represent the state-of-the-art approach for classifying empathy [1], [2] and generating empathetic responses [3], [4]. However, the development of robots capable of dynamically adjusting their level of empathy based on the context remains an underexplored area [5]. To this end, we introduce whEE (when and how to express empathy), an empathy framework that provides guidelines on when robots should respond empathetically and how to achieve it. Using our framework, we analyze the utterances of speakers and listeners in dyadic and group conversations with varying levels of empathy. Our analysis identifies key empathy cues that indicate when a speaker seeks an empathetic response and the cues exhibited by listeners displaying high levels of empathy. We approach empathy by focusing on observable behaviors that individuals exhibit when demonstrating an understanding of others' emotions and engaging deeply with their experiences--referred to as behavioral empathy [6].

Social robots aim to establish long-term bonds with humans through engaging conversation. However, traditional conversational approaches, reliant on scripted interactions, often fall short in maintaining engaging conversations. This paper addresses this limitation by integrating large language models (LLMs) into social robots to achieve more dynamic and expressive conversations. We introduce a fully-automated conversation system that leverages LLMs to generate robot responses with expressive behaviors, congruent with the robot's personality. We incorporate robot behavior with two modalities: 1) a text-to-speech (TTS) engine capable of various delivery styles, and 2) a library of physical actions for the robot. We develop a custom, state-of-the-art emotion recognition model to dynamically select the robot's tone of voice and utilize emojis from LLM output as cues for generating robot actions. A demo of our system is available here. To illuminate design and implementation issues, we conduct a pilot study where volunteers chat with a social robot using our proposed system, and we analyze their feedback, conducting a rigorous error analysis of chat transcripts. Feedback was overwhelmingly positive, with participants commenting on the robot's empathy, helpfulness, naturalness, and entertainment. Most negative feedback was due to automatic speech recognition (ASR) errors which had limited impact on conversations. However, we observed a small class of errors, such as the LLM repeating itself or hallucinating fictitious information and human responses, that have the potential to derail conversations, raising important issues for LLM application.

This study presents an empirical investigation into the design and impact of autonomous dialogues in human-robot interaction for behavior change coaching. We focus on the use of Haru, a tabletop social robot, and explore the implementation of the Tiny Habits method for fostering positive behavior change. The core of our study lies in developing a fully autonomous dialogue system that maximizes Haru's emotional expressiveness and unique personality. Our methodology involved iterative design and extensive testing of the dialogue system, ensuring it effectively embodied the principles of the Tiny Habits method while also incorporating strategies for trust-raising and trust-dampening. The effectiveness of the final version of the dialogue was evaluated in an experimental study with human participants (N=12). The results indicated a significant improvement in perceptions of Haru's liveliness, interactivity, and neutrality. Additionally, our study contributes to the broader understanding of dialogue design in social robotics, offering practical insights for future developments in the field.

Social robots have had it tough recently. There are lots of reasons for this, but a big part of it is that it's a challenge to develop a social robot that's able to spark long-term user interest without driving initial expectations impractically high. This isn't just the case for commercial robots--social robots designed for long-term user interaction studies have the same sorts of issues. The Honda Research Institute is well aware of how tricky this is, and researchers there have been working on the design of a prototype social robot that achieves a "balance between human expectation, surface appearance, physical affordance, and robot functionality." It's called Haru, and Honda Research has provided a fascinating and detailed look into how they came up with its design.