Versus Look Up Mashable's Best: E-readers, robovacs, laptops, earbuds, smart home and more Say More Creator Hub Gift Ideas For Everyone On Your List Mashable Selects Switch Off Trending Now Safety Net In My Bag VidCon with Mashable All Series Ahead of Made by Google, DeepMind revealed a new Pixel 11 feature that lets people sign instead of type. Olivia Tauber is the deputy editor of digital culture, covering creators, media, movies, beauty, and more. Based in New York, her work has appeared in The New York Times, Vanity Fair, The Cut, Teen Vogue, Complex, and Interview Magazine. She holds a Master's degree in Journalism from NYU and a Bachelor's from the University of Michigan. She also runs Fan Mail, a weekly pop-culture newsletter.

Smartphones have long offered voice-to-text transcription, but the same can't be said for sign-language-to-text. That's changing today with the launch of the Pixel 11 family, which marks the debut of DeepMind's new sign-language-to-text (SL2T) model. Google has embedded the model inside of Gboard and Live Transcribe, where it will allow deaf and hard of hearing users to sign to their phone anywhere where they would have normally typed in the past. In practice, that will give those users a faster, more natural way to search the web, write messages and converse with Google's Gemini chatbot. "There is great diversity among deaf people in terms of their level of proficiency in signing, speaking, reading, and writing, so it is important to support access in all modalities," says DeepMind.

Sign language translation (SL T) addresses the problem of translating information from a sign language in video to a spoken language in text. Existing studies, while showing progress, are often limited to narrow domains and/or few sign languages and struggle with open-domain tasks. In this paper, we push forward the frontier of SL T by scaling pretraining data, model size, and number of translation directions. We perform large-scale SL T pretraining on different data including 1) noisy multilingual Y ouTube SL T data, 2) parallel text corpora, and 3) SL T data augmented by translating video captions to other languages with off-the-shelf machine translation models. We unify different pretraining tasks with task-specific prompts under the encoder-decoder architecture, and initialize the SL T model with pretrained (m/By)T5 models across model sizes. SL T pretraining results on How2Sign and FLEURS-ASL#0 (ASL to 42 spoken languages) demonstrate the significance of data/model scaling and cross-lingual cross-modal transfer, as well as the feasibility of zero-shot SL T. We finetune the pretrained SL T models on 5 downstream open-domain SL T benchmarks covering 5 sign languages. Experiments show substantial quality improvements over the vanilla baselines, surpassing the previous state-of-the-art (SOT A) by wide margins.

Considering different geographic regions generally have their own native sign languages, it is valuable to establish corresponding SL T datasets to support related communication and research. Auslan, as a sign language specific to Australia, still lacks a dedicated large-scale dataset for SL T.

Communication in sign language is an essential part of many people's lives.