Figure 2: Testlog-perplexityofa Transformer -Bigmodelon WMT'14 en!fr, whentrainingwith batchsizesof 384 (left) and 768 (right).

Detailed ablation studies demonstrate thatself-attention isespecially impactful when used inlater layers. These results establish that stand-alone self-attention is an important addition to the vision practitioner'stoolbox.

Eight names are listed as authors on "Attention Is All You Need," a scientific paper written in the spring of 2017. They were all Google researchers, though by then one had left the company. When the most tenured contributor, Noam Shazeer, saw an early draft, he was surprised that his name appeared first, suggesting his contribution was paramount. "I wasn't thinking about it," he says. It's always a delicate balancing act to figure out how to list names--who gets the coveted lead position, who's shunted to the rear.

No conference on artificial intelligence (AI), machine learning or robotics would be complete without its fair share of technologists, programmers and engineers. But scan the list of attendees at the 2020 Rise of AI Summit, a hybrid (digital and physical) event this week in Berlin (November 17-18, 2020) and the number of people from health insurance companies, banks and venture capitalists is astonishing. As one of the founders of the event, CEO of Asgard Capital, Fabian Westerheide, said in his opening remarks on "The Next Decade of AI, we are in a'renaissance' of the technology." Westerheide says we're seeing a "refurbishment of ideas from the 1960s, 70s and 80s," combined with the amount of data we have now and today's processing power. He calls it "old ideas, new execution, and new capital."

In the fall of 2017, Sam Bowman, a computational linguist at New York University, figured that computers still weren't very good at understanding the written word. Sure, they had become decent at simulating that understanding in certain narrow domains, like automatic translation or sentiment analysis (for example, determining if a sentence sounds "mean or nice," he said). But Bowman wanted measurable evidence of the genuine article: bona fide, human-style reading comprehension in English. So he came up with a test. In an April 2018 paper coauthored with collaborators from the University of Washington and DeepMind, the Google-owned artificial intelligence company, Bowman introduced a battery of nine reading-comprehension tasks for computers called GLUE (General Language Understanding Evaluation). The test was designed as "a fairly representative sample of what the research community thought were interesting challenges," said Bowman, but also "pretty straightforward for humans." For example, one task asks whether a sentence is true based on information offered in a preceding sentence.

In the fall of 2017, Sam Bowman, a computational linguist at New York University, figured that computers still weren't very good at understanding the written word. Sure, they had become decent at simulating that understanding in certain narrow domains, like automatic translation or sentiment analysis (for example, determining if a sentence sounds "mean or nice," he said). But Bowman wanted measurable evidence of the genuine article: bona fide, human-style reading comprehension in English. So he came up with a test. In an April 2018 paper coauthored with collaborators from the University of Washington and DeepMind, the Google-owned artificial intelligence company, Bowman introduced a battery of nine reading-comprehension tasks for computers called GLUE (General Language Understanding Evaluation). The test was designed as "a fairly representative sample of what the research community thought were interesting challenges," said Bowman, but also "pretty straightforward for humans."