This year is likely to be remembered for the Covid-19 pandemic and for a significant presidential election, but there is a new contender for the most spectacularly newsworthy happening of 2020: the unveiling of GPT-3. As a very rough description, think of GPT-3 as giving computers a facility with words that they have had with numbers for a long time, and with images since about 2012. The core of GPT-3, which is a creation of OpenAI, an artificial intelligence company based in San Francisco, is a general language model designed to perform autofill. It is trained on uncategorized internet writings, and basically guesses what text ought to come next from any starting point. That may sound unglamorous, but a language model built for guessing with 175 billion parameters -- 10 times more than previous competitors -- is surprisingly powerful.

GPT is short for Generative Pre-training Transformer (GPT), a language model written by Alec Radford and published in 2018 by OpenAI, Elon Musks's artificial intelligence research laboratory. It uses a generative model of language (where two neural networks perfect each other by competition) and is able to acquire knowledge of the world and process long-range dependencies by pre-training on diverse sets of written material with long stretches of contiguous text. GPT-2 (Generative Pretrained Transformer 2) was announced in February 2019 and is an unsupervised transformer language model trained on 8 million documents for a total of 40 GB of text from articles shared via Reddit submissions. Elon Musk was famously reluctant to release it as he was concerned it could be used to spam social networks with fake news. In May 2020 OpenAI announced GPT-3 (Generative Pretrained Transformer 3), a model which contains two orders of magnitude more parameters than GPT-2 (175 billion vs 1.5 billion parameters) and which offers a dramatic improvement over GPT-2.

In 2017, researchers asked: Could AI write most code by 2040? OpenAI's GPT-3, now in use by beta testers, can already code in any language. Machine-dominated coding is almost at our doorstep. GPT-3 was trained on hundreds of billions of words, or essentially the entire Internet, which is why it can code in CSS, JSX, Python, -- you name it. Further, GPT-3 doesn't need to be "trained" for various language tasks, since its training data is all-encompassing.

In a study printed on the preprint server Arxiv.org, DeepMind researchers describe a reinforcement learning algorithm-generating approach that discovers what to foretell and the way to be taught it by interacting with environments. They declare the generated algorithms carry out nicely on a variety of difficult Atari video video games, reaching "non-trivial" efficiency indicative of the approach's generalizability. Reinforcement studying algorithms -- algorithms that allow software program brokers to be taught in environments by trial and error utilizing suggestions -- replace an agent's parameters in response to one in all a number of guidelines. These guidelines are often found via years of analysis, and automating their discovery from knowledge might result in extra environment friendly algorithms, or algorithms higher tailored to particular environments. DeepMind's answer is a meta-learning framework that collectively discovers what a specific agent ought to predict and the way to use the predictions for coverage enchancment.

And a tool like this has many new uses, both good (from powering better chatbots to helping people code) and bad (from powering better misinformation bots to helping kids cheat on their homework). But when a new AI milestone comes along it too often gets buried in hype. Even Sam Altman, who co-founded OpenAI with Elon Musk, tried to tone things down: "The GPT-3 hype is way too much. It's impressive (thanks for the nice compliments!) but it still has serious weaknesses and sometimes makes very silly mistakes. AI is going to change the world, but GPT-3 is just a very early glimpse. We have a lot still to figure out."

The artificial intelligence tool GPT-3 has been causing a stir online, due to its impressive ability to design websites, prescribe medication, and answer questions. GPT-3 is short for Generative Pre-training Transformer and is the third generation of the machine learning model. Machine learning is when computers can automatically learn from their experiences without having to be programmed. Its predecessor, GPT-2, made headlines for being deemed "too dangerous to release" because of its ability to create text that is seemingly indistinguishable from those written by humans. While GPT-2 had 1.5 billion parameters which could be set, GPT-3 has 175 billion parameters.

Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions – something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting.

OpenAI, a non-profit artificial intelligence research company backed by Peter Thiel, Elon Musk, Reid Hoffman, Marc Benioff, Sam Altman and others, released its third generation of language prediction model (GPT-3) into the open-source wild. Language models allow computers to produce random-ish sentences of approximately the same length and grammatical structure as those in a given body of text. In my early experiments with GPT-3 I found that GPT-3's predicted sentences, when published on the bitcointalk.org I imagine that similar results can be obtained by republishing GPT-3's outputs to other message boards, blogs, and social media. I predict that, unlike its two predecessors (PTB and OpenAI GPT-2), OpenAI GPT-3 will eventually be widely used to pretend the author of a text is a person of interest, with unpredictable and amusing effects on various communities.

The introduction of transfer learning and pretrained language models in natural language processing (NLP) pushed forward the limits of language understanding and generation. Transfer learning and applying transformers to different downstream NLP tasks have become the main trend of the latest research advances. At the same time, there is a controversy in the NLP community regarding the research value of the huge pretrained language models occupying the leaderboards. While lots of AI experts agree with Anna Rogers's statement that getting state-of-the-art results just by using more data and computing power is not research news, other NLP opinion leaders point out some positive moments in the current trend, like, for example, the possibility of seeing the fundamental limitations of the current paradigm. Anyway, the latest improvements in NLP language models seem to be driven not only by the massive boosts in computing capacity but also by the discovery of ingenious ways to lighten models while maintaining high performance.

It's easy to imagine the artificial intelligence (AI) revolution as years off in the future, but it might be much closer than you think. When Sharif Shameem posted to Twitter an experiment he did with GPT-3, a closed-access artificial intelligence, thousands in the technology community were stunned. With GPT-3, I built a layout generator where you just describe any layout you want, and it generates the JSX code for you. With seemingly little effort, the two-minute clip appeared to show an AI understand how to write fairly complex computer code from a request in plain English, despite never having been trained to write code in the first place – or even understand English. 'It was never explicitly programmed how to read or how to understand English,' says Shameem, who has founded a start-up to help people develop web applications by writing in plain English.