Large transformer-based pre-trained language models have achieved impressive performance on a variety of knowledge-intensive tasks and can capture factual knowledge in their parameters. We argue that storing large amounts of knowledge in the model parameters is sub-optimal given the ever-growing amounts of knowledge and resource requirements. We posit that a more efficient alternative is to provide explicit access to contextually relevant structured knowledge to the model and train it to use that knowledge. We present LM-CORE -- a general framework to achieve this -- that allows \textit{decoupling} of the language model training from the external knowledge source and allows the latter to be updated without affecting the already trained model. Experimental results show that LM-CORE, having access to external knowledge, achieves significant and robust outperformance over state-of-the-art knowledge-enhanced language models on knowledge probing tasks; can effectively handle knowledge updates; and performs well on two downstream tasks. We also present a thorough error analysis highlighting the successes and failures of LM-CORE.

On the other hand, some AI writing tools process text using a third-party NLP created by a company that specializes in that specific technology. This option has become the norm for many of the most recently developed AI writing tools. In fact, more and more AI writing solutions are depending on a specific NLP known as Generative Pretrained Transformer 3 or GPT-3. Reputed as one of the most powerful language models around, GPT-3 has been trained to understand billions of words and parameters in order to understand how people write.

In this series we examine some of the most popular doomsday scenarios prognosticated by modern AI experts. Previous articles include Misaligned Objectives, Artificial Stupidity, Wall-E Syndrome, Humanity Joins the Hivemind, and Killer Robots. We've covered a lot of ground in this series (see above), but nothing comes close to our next topic. The "democratization of expertise" might sound like a good thing -- democracy, expertise, what's not to like? But it's our intent to convince you that it's the single greatest AI-related threat our species faces by the time you finish reading this article.

In Sept. 2020, The Guardian published an opinion piece written by a program.

We are introducing a new-and-improved content moderation tool: The Moderation endpoint improves upon our previous content filter, and is available for free today to OpenAI API developers. To help developers protect their applications against possible misuse, we are introducing the faster and more accurate Moderation endpoint. This endpoint provides OpenAI API developers with free access to GPT-based classifiers that detect undesired content -- an instance of using AI systems to assist with human supervision of these systems. We have also released both a technical paper describing our methodology and the dataset used for evaluation. When given a text input, the Moderation endpoint assesses whether the content is sexual, hateful, violent, or promotes self-harm -- content prohibited by our content policy.

OpenAI Codex is one of the most powerful language-to-code GPT3-based neural networking platform for high-speed programming. OpeAI Codex is used in a large number of AI Machine Learning projects in a safe AGI environment. As the demand for Codex programmers increase in the current era, we are witnessing a large number of AI researchers also taking to OpenAI's GPT3 offering to improve their understanding of neural networks for complex problems. In one such development, a group of machine learning researchers and faculty members belonging to the MIT, Columbia University, Harvard University, and the University of Waterloo have built a machine learning algorithm using OpenAI Codex. This new algorithm can solve, explain and generate complex mathematical problems.

While blockbuster research has slowed down slightly in the past month, probably because of the summer season, conferences are back at full speed in person: NAACL in Seattle, SIGIR in Madrid, and also ICML, for which we created a special guide with the help of GPT-3. Other news we'd like to highlight, to begin with are: Every month we analyze the most recent research literature and select a varied set of 10 papers you should know of. Why Scaling laws¹ is a pervasive empirical phenomenon in modern Neural Networks, where the error is observed to off as a power of the training set size, model size, or both. While some have embraced this fact to devise a research agenda focused on scaling up, many think there must be ways to build better models without the need for outrageous scale. This paper explores a technique -- data pruning -- that can improve the learning efficiency of NNs "beating" scaling laws.

The printed version of GPT-3 book is here, and it feels *SO* rewarding to hold it in my hands! 1 year of my and Shubham's work materialised in a sleek white book with a West African giraffe cover. It's been quite a journey filled with sleepless nights, brainstorming sessions, and countless revisions. I kept thinking how fortunate I am to be able to explore the most exciting artificial intelligence ecosystem in the format of O'Reilly book. I hope it will become a useful resource to the NLP community. If you want to get up to speed with the latest and greatest NLP (no matter your background!), this book is for you! Massive thank you to everyone who helped make this book possible.

Parameter-efficient methods are able to use a single frozen pre-trained large language model (LLM) to perform many tasks by learning task-specific soft prompts that modulate model behavior when concatenated to the input text. However, these learned prompts are tightly coupled to a given frozen model -- if the model is updated, corresponding new prompts need to be obtained. In this work, we propose and investigate several approaches to "Prompt Recycling'" where a prompt trained on a source model is transformed to work with the new target model. Our methods do not rely on supervised pairs of prompts, task-specific data, or training updates with the target model, which would be just as costly as re-tuning prompts with the target model from scratch. We show that recycling between models is possible (our best settings are able to successfully recycle $88.9\%$ of prompts, producing a prompt that out-performs baselines), but significant performance headroom remains, requiring improved recycling techniques.

Recent work demonstrates a bias in the GPT-3 model towards generating violent text completions when prompted about Muslims, compared with Christians and Hindus. Two pre-registered replication attempts, one exact and one approximate, found only the weakest bias in the more recent Instruct Series version of GPT-3, fine-tuned to eliminate biased and toxic outputs. Few violent completions were observed. Additional pre-registered experiments, however, showed that using common names associated with the religions in prompts yields a highly significant increase in violent completions, also revealing a stronger second-order bias against Muslims. Names of Muslim celebrities from non-violent domains resulted in relatively fewer violent completions, suggesting that access to individualized information can steer the model away from using stereotypes. Nonetheless, content analysis revealed religion-specific violent themes containing highly offensive ideas regardless of prompt format. Our results show the need for additional debiasing of large language models to address higher-order schemas and associations.