The Transform Technology Summits start October 13th with Low-Code/No Code: Enabling Enterprise Agility. In January 2020, OpenAI laid out the scaling law of language models: You can improve the performance of any neural language model by adding more training data, more model parameters, and more compute. Since then, there has been an arms race to train ever larger neural networks for natural language processing (NLP). And the latest to join the list is AI21 with its 178 billion parameter model. Before this, Amnon founded Mobileye, the NYSE-listed self-driving tech company that Intel acquired for $15.4 billion.

AI21 Labs: OpenAI's GPT-3 is the better part of a year and remained among the largest Artificial Intelligence system in the terms of language models which is ever been created or came into existence. With the help of an API, it has become so easy to use that people are using it for automatically writing the articles and emails along with summarizing the texts, composition of poetries and recipes, generating the codes for deep learning in Python, and creating layouts and templates for websites. But now an Artificial Intelligence lab is based in Tel Aviv, Israel which is named AI21 Labs which stated that they are planning to release a larger model and make it available via a service with the idea of being challenged by OpenAI's dominance in the Natural Language Processing as a service for the development of the Artificial Intelligence field. The startup stated that the largest version of their Artificial Intelligence model is known as Jurassic-1 Jumbo which contains 178 billion parameters and more than 3 billion GPT-3. Taking a look towards Artificial Intelligence along with machine learning parameters are the most important part of the model that is learned from historical training data.

The goal of Multi-task Bayesian Optimization (MBO) is to minimize the number of queries required to accurately optimize a target black-box function, given access to offline evaluations of other auxiliary functions. When offline datasets are large, the scalability of prior approaches comes at the expense of expressivity and inference quality. We propose JUMBO, an MBO algorithm that sidesteps these limitations by querying additional data based on a combination of acquisition signals derived from training two Gaussian Processes (GP): a cold-GP operating directly in the input domain and a warm-GP that operates in the feature space of a deep neural network pretrained using the offline data. Such a decomposition can dynamically control the reliability of information derived from the online and offline data and the use of pretrained neural networks permits scalability to large offline datasets. Theoretically, we derive regret bounds for JUMBO and show that it achieves no-regret under conditions analogous to GP-UCB (Srinivas et. al. 2010). Empirically, we demonstrate significant performance improvements over existing approaches on two real-world optimization problems: hyper-parameter optimization and automated circuit design.