Socially Interactive Agents (SIAs) are physical or virtual embodied agents that display similar behavior as human multimodal behavior. Modeling SIAs' non-verbal behavior, such as speech and facial gestures, has always been a challenging task, given that a SIA can take the role of a speaker or a listener. A SIA must emit appropriate behavior adapted to its own speech, its previous behaviors (intra-personal), and the User's behaviors (inter-personal) for both roles. We propose AMII, a novel approach to synthesize adaptive facial gestures for SIAs while interacting with Users and acting interchangeably as a speaker or as a listener. AMII is characterized by modality memory encoding schema - where modality corresponds to either speech or facial gestures - and makes use of attention mechanisms to capture the intra-personal and inter-personal relationships. We validate our approach by conducting objective evaluations and comparing it with the state-of-the-art approaches.

Google is offering up new ways for people with various disabilities to more easily use their smartphones. The technology giant said that it's now possible to control Android-powered smartphones hands-free by using simple gestures like smiling, raising eyebrows or looking to one direction. The new options are available through two tools -- Camera Switches and Project Activate -- which use machine learning technology and the phone's front-facing camera to pick up on gestures. Camera Switches is a feature inside Switch Access, a long-standing option allowing people to use adaptive buttons called physical switches to control their phones. The new offering allows users to assign six types of facial gestures to different actions like returning to the home screen or opening notifications.

Using a raised eyebrow or smile, people with speech or physical disabilities can now operate their Android-powered smartphones hands-free, Google said Thursday. Two new tools put machine learning and front-facing cameras on smartphones to work detecting face and eye movements. Users can scan their phone screen and select a task by smiling, raising eyebrows, opening their mouth or looking to the left, right or up. "To make Android more accessible for everyone, we're launching new tools that make it easier to control your phone and communicate using facial gestures," Google said. The Centers for Disease Control and Prevention estimates that 61 million adults in the United States live with disabilities, which has pushed Google and rivals Apple and Microsoft to make products and services more accessible to them.

To enable more natural face-to-face interactions, conversational agents need to adapt their behavior to their interlocutors. One key aspect of this is generation of appropriate non-verbal behavior for the agent, for example facial gestures, here defined as facial expressions and head movements. Most existing gesture-generating systems do not utilize multi-modal cues from the interlocutor when synthesizing non-verbal behavior. Those that do, typically use deterministic methods that risk producing repetitive and non-vivid motions. In this paper, we introduce a probabilistic method to synthesize interlocutor-aware facial gestures - represented by highly expressive FLAME parameters - in dyadic conversations. Our contributions are: a) a method for feature extraction from multi-party video and speech recordings, resulting in a representation that allows for independent control and manipulation of expression and speech articulation in a 3D avatar; b) an extension to MoGlow, a recent motion-synthesis method based on normalizing flows, to also take multi-modal signals from the interlocutor as input and subsequently output interlocutor-aware facial gestures; and c) a subjective evaluation assessing the use and relative importance of the input modalities. The results show that the model successfully leverages the input from the interlocutor to generate more appropriate behavior. Videos, data, and code available at: https://jonepatr.github.io/lets_face_it.

A new wheelchair may give people with severe mobility challenges another reason to smile about artificial intelligence--that grin might literally help them control their wheelchair. Sao Paulo, Brazil-based Hoobox Robotics has teamed up with Intel on the Wheelie 7, a kit that leverages AI to let a disabled person drive a motorized wheelchair through any of 10 facial expressions, from raising an eyebrow to sticking out one's tongue. Motorized wheelchairs these days are typically controlled with a user's hands, a joystick or via sensors attached to the body. The Wheelie learns the user's smile and other gestures automatically--there is no special training that is required. Through an app, a caregiver or family member can assign which facial expressions would be tied to which way the wheelchair moves or stops: left, right, forward, backwards.

People feel emotions when listening to music. However, emotions are not tangible objects that can be exploited in the music composition process as they are difficult to capture and quantify in algorithms. We present a novel musical interface, Mugeetion, designed to capture occurring instances of emotional states from users' facial gestures and relay that data to associated musical features. Mugeetion can translate qualitative data of emotional states into quantitative data, which can be utilized in the sound generation process. We also presented and tested this work in the exhibition of sound installation, Hearing Seascape, using the audiences' facial expressions. Audiences heard changes in the background sound based on their emotional state. The process contributes multiple research areas, such as gesture tracking systems, emotion-sound modeling, and the connection between sound and facial gesture.

When a proud parent coos, or pokes out their tongue it warms their heart to see their baby following suit and many believe their little darling is copying them as they peer over the cot. But new research has found that babies up to the age of two months are incapable of copying facial expressions, gestures or speech. Instead, any exaggerated movements the newborns make are simply because they are responding to excitement to the interaction. When a proud parent coos, or pokes out their tongue, it warms their heart to see their baby following suit and many believe their little darling is copying them as they peer over the cot. While it may look like they are imitating the example of their elders, they are making gestures they would have made anyway.