Nonverbal Communication through Expressive Objects

Augmentative and alternative communication (AAC) devices enable speech-based communication, but generating speech is not the only resource needed to have a successful conversation. Being able to signal one wishes to take a turn by raising a hand or providing some other cue is critical in securing a turn to speak. Experienced conversation partners know how to recognize the nonverbal communication an augmented communicator (AC) displays, but these same nonverbal gestures can be hard to interpret by people who meet an AC for the first time. Prior work has identified motion through robots and expressive objects as a modality that can support communication. In this work, we work closely with an AAC user to understand how motion through a physical expressive object can support their communication. We present our process and resulting lessons on the designed object and the co-design process. Augmented communicators (ACs) with motor disabilities that affect speech production may use augmentative and alternative communication (AAC) devices to speak. AAC devices include picture or letter boards that people can point to or speech-generating devices people can use to compose messages.2 Commercial speech-generating AAC systems are currently only customizable at the word selection and speech production levels, and they do not yet support augmentations that can increase non-verbal communication. Nonverbal communication is key in helping regulate turn-taking, convey personality, and execute actions that increase social agency,12 all of which are current challenges for ACs.15, 22 For instance, ACs are compelled to respond within the synchronous timing constraints of in-person interactions even though they use an asynchronous text-based medium.10 ACs have to compose a message on their device using text and then they share their message with text-to-speech while a non-augmented conversation partner responds synchronously using speech without needing to compose a message. Prior work identified motion-based AAC as a viable and under-explored modality for increasing ACs' agency in conversation.21 We build on this prior-work to dig deeper into a particular case study on motion-based AAC by co-designing a physical expressive object, or sidekick, to support ACs during conversations.