Language models (LMs) exhibit remarkable abilities to solve new tasks from just a few examples or textual instructions, especially at scale. They also, paradoxically, struggle with basic functionality, such as arithmetic or factual lookup, where much simpler and smaller models excel. In this paper, we show that LMs can teach themselves to use external tools via simple APIs and achieve the best of both worlds. We introduce Toolformer, a model trained to decide which APIs to call, when to call them, what arguments to pass, and how to best incorporate the results into future token prediction. This is done in a self-supervised way, requiring nothing more than a handful of demonstrations for each API. We incorporate a range of tools, including a calculator, a Q\&A system, two different search engines, a translation system, and a calendar. Toolformer achieves substantially improved zero-shot performance across a variety of downstream tasks, often competitive with much larger models, without sacrificing its core language modeling abilities.

Specialised pre-trained language models are becoming more frequent in NLP since they can potentially outperform models trained on generic texts. BioBERT (Sanh et al., 2019) and BioClinicalBERT (Alsentzer et al., 2019) are two examples of such models that have shown promise in medical NLP tasks. Many of these models are overparametrised and resource-intensive, but thanks to techniques like Knowledge Distillation (KD), it is possible to create smaller versions that perform almost as well as their larger counterparts. In this work, we specifically focus on development of compact language models for processing clinical texts (i.e. We developed a number of efficient lightweight clinical transformers using knowledge distillation and continual learning, with the number of parameters ranging from 15 million to 65 million. These models performed comparably to larger models such as BioBERT and ClinicalBioBERT and significantly outperformed other compact models trained on general or biomedical data. Our extensive evaluation was done across several standard datasets and covered a wide range of clinical text-mining tasks, including Natural Language Inference, Relation Extraction, Named Entity Recognition, and Sequence Classification. To our knowledge, this is the first comprehensive study specifically focused on creating efficient and compact transformers for clinical NLP tasks. The models and code used in this study can be found on our Huggingface profile at https: //huggingface.co/nlpie and Github page at https://github.com/ Large language models pre-trained on generic texts serve as the foundation upon which most stateof-the-art NLP models are built. There is ample evidence that, for certain domains and downstream tasks, models that are pre-trained on specialised data outperform baselines that have only relied on generic texts (Sanh et al., 2019; Alsentzer et al., 2019; Beltagy et al., 2019; Nguyen et al., 2020; Chalkidis et al., 2020).

Fine-tuning the entire set of parameters of a large pretrained model has become the mainstream approach for transfer learning. To increase its efficiency and prevent catastrophic forgetting and interference, techniques like adapters and sparse fine-tuning have been developed. Adapters are modular, as they can be combined to adapt a model towards different facets of knowledge (e.g., dedicated language and/or task adapters). Sparse fine-tuning is expressive, as it controls the behavior of all model components. In this work, we introduce a new fine-tuning method with both these desirable properties. In particular, we learn sparse, real-valued masks based on a simple variant of the Lottery Ticket Hypothesis. Task-specific masks are obtained from annotated data in a source language, and language-specific masks from masked language modeling in a target language. Both these masks can then be composed with the pretrained model. Unlike adapter-based fine-tuning, this method neither increases the number of parameters at inference time nor alters the original model architecture. Most importantly, it outperforms adapters in zero-shot cross-lingual transfer by a large margin in a series of multilingual benchmarks, including Universal Dependencies, MasakhaNER, and AmericasNLI. Based on an in-depth analysis, we additionally find that sparsity is crucial to prevent both 1) interference between the fine-tunings to be composed and 2) overfitting. We release the code and models at https://github.com/cambridgeltl/composable-sft.

Incident management for cloud services is a complex process involving several steps and has a huge impact on both service health and developer productivity. On-call engineers require significant amount of domain knowledge and manual effort for root causing and mitigation of production incidents. Recent advances in artificial intelligence has resulted in state-of-the-art large language models like GPT-3.x (both GPT-3.0 and GPT-3.5), which have been used to solve a variety of problems ranging from question answering to text summarization. In this work, we do the first large-scale study to evaluate the effectiveness of these models for helping engineers root cause and mitigate production incidents. We do a rigorous study at Microsoft, on more than 40,000 incidents and compare several large language models in zero-shot, fine-tuned and multi-task setting using semantic and lexical metrics. Lastly, our human evaluation with actual incident owners show the efficacy and future potential of using artificial intelligence for resolving cloud incidents.

Tools like ChatGPT and DALL-E 2 (text-to-text or text-to-image AI tools) are all the rage these days. But for them to work effectively, you need to ask the right questions to get the results you want. Learning what to say to these tools will only become more important as they become more integrated in various industries. Check out Unite.ai's very own generative AI: Images.ai AI prompt engineering is an effective way to get the desired output with an AI tool.

Three months ago, ChatGPT debuted--the first artificial-intelligence bot to produce original content virtually indistinguishable from that of a human brain. Now, the creators of that software are beta testing a new tool that can (or so they say) determine whether a text was written by a person or a machine. The applications could be many, from identifying disinformation campaigns to detecting when a job candidate is has used AI for a cover letter. But experts worry the software will only create more challenges for leaders already caught in an AI rabbit hole. "The mushing of original thinking and discernment and artificial intelligence is dangerous for employees, managers, and leaders," says Andrés Tapia, a senior client partner and global diversity, equity, and inclusion strategist at Korn Ferry.

If it seems like everyone is rushing to develop an alternative to ChatGPT, you're not wrong. Chinese online commerce heavyweight Alibaba has confirmed to CNBC that it's working on its equivalent to OpenAI's hit AI chatbot. The company isn't detailing features or offering a release schedule, but says it has been developing generative AI since 2017 and is in the middle of internal testing. The reveal comes as multiple tech giants have introduced rivals to or spinoffs of ChatGPT this week. Google unveiled Bard, while China's Baidu said it was testing "Ernie Bot." Microsoft, meanwhile, launched a redesigned Bing that uses a "much more powerful" language model built with OpenAI's help.

A product race is under way in the world of artificial intelligence. Just this week, Google announced plans to release Bard, a search chatbot based on its proprietary large language model; yesterday, Microsoft held an event unveiling a next-generation web browser with a supercharged Bing interface powered by ChatGPT. Though most big tech companies have been quietly developing their own generative-AI tools for years, these giants are scrambling to demonstrate their chops after the public release and runaway adoption of OpenAI's ChatGPT, which has accumulated more than 30 million users in two months. OpenAI's success is an apparent signal to tech leaders that deep-learning networks are the next frontier of the commercial internet. AI evangelists will similarly tell you that generative AI is destined to become the overlay for not only search engines, but also creative work, busywork, memo writing, research, homework, sketching, outlining, storyboarding, and teaching.

If you've been following tech news at all for the last few months, you've no doubt heard that AI-powered text generation is kind of a Big Deal. ChatGPT and its alternatives are beginning to change the way we do a lot of things, like generating lists of instructions, planning vacations, or cheating at school essays. Microsoft has decided to dive in head first, debuting a new version of its Bing search engine with GPT-powered answers. You can try it right now. ChatGPT still requires a huge amount of processing power to run compared to standard search engines, so it's unlocking in a timed manner.

Bing's new AI-powered chatbot is basically ChatGPT with ads…and one that refuses to do your homework for you. That's not necessarily a derogatory criticism; the new Bing is pretty amazing. In your first few minutes with the new Bing chat interface, you'll probably see even more sophistication than the free version of ChatGPT currently offers, with lengthy, detailed responses that can help you in many walks of life. But they may end with a jarring ad that looks (and probably is) ripped straight from Bing. That said, the fresh AI experience already work shockingly well more often than not.