The fervor around state-of-the-art AI language models like OpenAI's GPT-3 hasn't died down. Melanie Mitchell, a professor of computer science at Portland State University, found evidence that GPT-3 can make primitive analogies. Raphaël Millière, a philosopher of mind and cognitive science at Columbia University's Center for Science and Society, asked GPT-3 to compose a response to the philosophical essays written about it. Among other applications, the API providing access to the model has been used to create a recipe generator, an all-purpose Excel function, and a comedy sketch writer. But even language models as powerful as GPT-3 have limitations that remain unaddressed.

The fervor around state-of-the-art AI language models like OpenAI's GPT-3 hasn't died down. Melanie Mitchell, a professor of computer science at Portland State University, found evidence that GPT-3 can make primitive analogies. Raphaël Millière, a philosopher of mind and cognitive science at Columbia University's Center for Science and Society, asked GPT-3 to compose a response to the philosophical essays written about it. Among other applications, the API providing access to the model has been used to create a recipe generator, an all-purpose Excel function, and a comedy sketch writer. But even language models as powerful as GPT-3 have limitations that remain unaddressed.

Disclaimer: My opinions are informed by my experience maintaining Cortex, an open source platform for machine learning engineering. If you frequent any part of the tech internet, you've come across GPT-3, OpenAI's new state of the art language model. While hype cycles forming around new technology isn't new--GPT-3's predecessor, GPT-2, generated quite a few headlines as well--GPT-3 is in a league of its own. If you're on Twitter, you've no doubt seen projects built on GPT-3 going viral, like this Apple engineer who used GPT-3 to write Javascript using a specific 3D rendering library: And of course, there have been plenty of "Is this the beginning of SkyNet?" articles written: The excitement over GPT-3 is just a piece of an bigger trend. Every month, we see more and more new initiatives release, all built on machine learning.

GPT-3 is a language model that is automated by a neural system, launched by OpenAI in July 2020. It's a text generator that can compose articles, poetry, sentiment essays, and working code--which is the reason it has the entire world humming, some with excitement, some with skepticism. The previous GPT model had 1.5 billion parameters and was the biggest model in those days, which was before long overshadowed by NVIDIA's Megatron, with 8 billion parameters followed by Microsoft's Turing NLG that had 17 billion parameters. Presently, OpenAI changes the situation by deploying a model that is 10 times bigger than Turing NLG. Current NLP frameworks still to a great extent struggle to learn from a couple of models.

In a paper published on the preprint server Arxiv.org, This alone isn't surprising, but the problem is that data from these communities are often used to train large language models like OpenAI's GPT-3. That in turn is important because, as OpenAI itself notes, this sort of bias leads to placing words like "naughty" or "sucked" near female pronouns and "Islam" near words like "terrorism." The scientists' approach uses representations of words called embeddings to discover and categorize language biases, which could enable data scientists to trace the severity of bias in different communities and take steps to counteract this bias. To spotlight examples of potentially offensive content on Reddit subcommunities, given a language model and two sets of words representing concepts to compare and discover biases from, the method identifies the most biased words toward the concepts in a given community.

The SEC said, "Musk,/your tweets are a blight./They Musk cried, "Why?/The tweets I wrote are not mean,/I don't use all-caps/and I'm sure that my tweets are clean."/"But THE PRECEDING lines--describing Tesla and SpaceX founder Elon Musk's run-ins with the Securities and Exchange Commission, an American financial regulator--are not the product of some aspiring 21st-century Dr Seuss. They come from a poem written by a computer running a piece of software called Generative Pre-Trained Transformer 3. GPT-3, as it is more commonly known, was developed by OpenAI, an artificial-intelligence (AI) laboratory based in San Francisco, and which Mr Musk helped found. It represents the latest advance in one of the most studied areas of AI: giving computers the ability to generate sophisticated, human-like text.

OpenAI's latest language generation model, GPT-3, has made quite the splash within AI circles, astounding reporters to the point where even Sam Altman, OpenAI's leader, mentioned on Twitter that it may be overhyped. Still, there is no doubt that GPT-3 is powerful. Those with early-stage access to OpenAI's GPT-3 API have shown how to translate natural language into code for websites, solve complex medical question-and-answer problems, create basic tabular financial reports, and even write code to train machine learning models -- all with just a few well-crafted examples as input (i.e., via "few-shot learning"). Soon, anyone will be able to purchase GPT-3's generative power to make use of the language model, opening doors to build tools that will quietly (but significantly) shape our world. Enterprises aiming to take advantage of GPT-3, and the increasingly powerful iterations that will surely follow, must take great care to ensure that they install extensive guardrails when using the model, because of the many ways that it can expose a company to legal and reputational risk.

OpenAI's GPT-3 is the talk of the town, and the media is giving it all the attention. Many analysts are even comparing it to AGI because of its practical applicability. Initially disclosed in a research paper in May, GPT-3 is the next version of GPT-2 and is 100x larger than it. It is far more competent than its forerunner due to the number of parameters it is trained on, which is 175 billion for GPT-3 versus 1.5 billion for GPT-2. After the successful launch of GPT-3, other AI companies seem to have been overshadowed.

What does it mean when a computer can write about our problems better than we can? People have been talking a lot about GPT-3, but more as a novelty than a tool (don't know what GPT-3 is? look here). Some clever people have even figured out how to get it to generate code from descriptions. Yet, I think that the best use cases lie outside of tech. I believe that GPT-3 has the potential to change the way we write.

Psychological semantics is the study of how people represent the meanings of words and then build sentence meaning out of those representations. People use language dozens of time a day--to have conversations and give instructions, to read and write, to label objects and teach. A theory of psychological semantics must provide the basis for how people do all those things, choosing which words to use and understanding the words they read or hear. In this article we focus on the mental representation of word meaning. Human language is still the gold standard for a communication system, but artificial intelligence (AI) systems have made important progress in language use. Research on Natural Language Processing (NLP) develops systems that understand language to the degree that computers can carry out useful tasks. As described below, such systems use vast text corpora to learn about words, using neural networks and other statistical models. The recent explosion of research in NLP, driven largely by advances in neural networks (also called deep learning), has resulted in continuously improving performance on various benchmarks that require interpreting words and sentences. Systems are now used in interfaces with customers to make sales or solve problems.