Your cell phone can handle basic language translation, but bespoke tools can offer a much more immersive experience. Hans Christian Andersen once said, "To travel is to live," and while that's a romantic notion, he probably wasn't careening through Gyeongju, South Korea, at midnight in the back of a taxi with a driver who didn't speak a lick of English. Today's world traveler has it awfully easy when it comes to understanding the local lingo, as even a basic modern cell phone app can offer a pretty good translation of common phrases delivered in everything from Abkhaz to Zulu. Type or speak a sentence or two into the app, tap a button, and out it returns in the language of your choice. Tap another button, and your phone can even speak those sentences aloud.

We then train and release a 10.7B-parameter

Neural Information Processing Systems http://nips.cc/

Awordvectorspace-sometimes referred toasawordembedding-associates similar wordsina vocabularywithsimilar vectors.

After annotating the entire dataset in each language, there was an additional annotator for each language who reviewed the entire set. Annotators were volunteers, and theyare acknowledged at theendofthiswork.

The Transformer model has achieved state-of-the-art performance in many sequence modeling tasks. However, how to leverage model capacity with large or variable depths is still an open challenge. We present a probabilistic framework to automatically learn which layer(s) to use by learning the posterior distributions of layer selection. As an extension of this framework, we propose a novel method to train one shared Transformer network for multilingual machine translation with different layer selection posteriors for each language pair. The proposed method alleviates the vanishing gradient issue and enables stable training of deep Transformers (e.g. 100 layers). We evaluate on WMT English-German machine translation and masked language modeling tasks, where our method outperforms existing approaches for training deeper Transformers. Experiments on multilingual machine translation demonstrate that this approach can effectively leverage increased model capacity and bring universal improvement for both many-to-one and one-to-many translation with diverse language pairs.