If you have trouble reading this email, see it on a web browser. Work in the AI field is moving forward very quickly. Today Papers with Code announced their partnership with arXiv, where code links are now shown on arXiv articles, and authors can submit code through arXiv, making it a great addition to avid researchers and practitioners. NeurIPS also announced a cool challenge, the 2020 ML Reproducibility Challenge sponsored by Papers with Code, encouraging people who work with ML to participate (including enthusiasts!). If you'd like to learn more, check out their announcement, it sounds pretty neat.

Every one is talking about the mighty, great, futuristic language model by OpenAI, founded by Tesla CEO Elon Musk, Y Combinator partner Sam Altman and other Silicon Valley big shots like Google researchers and ex CTO of Stripe. It is truly eye opening. We also want to tell you how exciting it is. Why is GPT-3 so hyped right now? Probably because GPT-3 has the coolest video demos ever: based on just a few English sentences it can generate a TODO app (write code by itself), generate Excel spreadsheets, automatically translate, generate quizzes based on content. Every one is writing about GPT-3, but because we are technical, our article will give you the important technical details and background you need to understand OpenAI's GPT-3.

For more than four decades, University of California, San Diego, Professor of Physics Patrick H. Diamond and his research group have been advancing our understanding of fundamental concepts in plasma physics. Most recently, Diamond worked with graduate student Robin Heinonen on a model reduction study that used the Extreme Science and Engineering Discovery Environment (XSEDE)-allocated Comet supercomputer at the San Diego Supercomputer Center at UC San Diego to showcase how machine learning produced a new model for plasma turbulence. Plasmas have many applications, including fusion energy. When light nuclei fuse together, the mass of the products is less than that of the reactants, and the missing mass becomes energy – hence Albert Einstein's famous E mc2 equation. In order for this to occur, temperatures must literally reach astronomical levels, such as those found in the Sun's core.

When you take a medication, you want to know precisely what it does. Pharmaceutical companies go through extensive testing to ensure that you do. With a new deep learning-based technique created at Rice University's Brown School of Engineering, they may soon get a better handle on how drugs in development will perform in the human body. The Rice lab of computer scientist Lydia Kavraki has introduced Metabolite Translator, a computational tool that predicts metabolites, the products of interactions between small molecules like drugs and enzymes. The Rice researchers take advantage of deep-learning methods and the availability of massive reaction datasets to give developers a broad picture of what a drug will do.

In this episode of the Data Exchange I speak with Murat Özbayoğlu, Chair of Artificial Intelligence Engineering at TOBB University of Economics and Technology in Ankara, Turkey. I've long been fascinated with finance and trading. My first job after I left academia was as the lead quant in a hedge fund, and ever since, I've tried to stay abreast of what tools and techniques quants and data scientists in finance are using. Forecasting in this setting usually means price prediction or price movement (trend) prediction. Output of forecasting models are used to inform investment decisions.

Forecasting complex dynamical phenomena in settings where only partial knowledge of their dynamics is available is a prevalent problem across various scientific fields. While purely data-driven approaches are arguably insufficient in this context, standard physical modeling based approaches tend to be over-simplistic, inducing non-negligible errors. In this work, we introduce the APHYNITY framework, a principled approach for augmenting incomplete physical dynamics described by differential equations with deep data-driven models. It consists in decomposing the dynamics into two components: a physical component accounting for the dynamics for which we have some prior knowledge, and a data-driven component accounting for errors of the physical model. The learning problem is carefully formulated such that the physical model explains as much of the data as possible, while the data-driven component only describes information that cannot be captured by the physical model, no more, no less. This not only provides the existence and uniqueness for this decomposition, but also ensures interpretability and benefits generalization. Experiments made on three important use cases, each representative of a different family of phenomena, i.e. reaction-diffusion equations, wave equations and the nonlinear damped pendulum, show that APHYNITY can efficiently leverage approximate physical models to accurately forecast the evolution of the system and correctly identify relevant physical parameters. Modeling and forecasting complex dynamical systems is a major challenge in domains such as environment and climate (Rolnick et al., 2019), health science (Choi et al., 2016), and in many industrial applications (Toubeau et al., 2018). Model Based (MB) approaches typically rely on partial or ordinary differential equations (PDE/ODE) and stem from a deep understanding of the underlying physical phenomena.

Continual learning could shift the machine learning paradigm from data centric to model centric. A continual learning model needs to scale efficiently to handle semantically different datasets, while avoiding unnecessary growth. We introduce hash-routed convolutional neural networks: a group of convolutional units where data flows dynamically. Feature maps are compared using feature hashing and similar data is routed to the same units. A hash-routed network provides excellent plasticity thanks to its routed nature, while generating stable features through the use of orthogonal feature hashing. Each unit evolves separately and new units can be added (to be used only when necessary). Hash-routed networks achieve excellent performance across a variety of typical continual learning benchmarks without storing raw data and train using only gradient descent. Besides providing a continual learning framework for supervised tasks with encouraging results, our model can be used for unsupervised or reinforcement learning.

Task-oriented dialogue systems help users accomplish tasks such as booking a movie ticket and ordering food via conversation. Generative models parameterized by a deep neural network are widely used for next turn response generation in such systems. It is natural for users of the system to want to accomplish multiple tasks within the same conversation, but the ability of generative models to compose multiple tasks is not well studied. In this work, we begin by studying the effect of training human-human task-oriented dialogues towards improving the ability to compose multiple tasks on Transformer generative models. To that end, we propose and explore two solutions: (1) creating synthetic multiple task dialogue data for training from human-human single task dialogue and (2) forcing the encoder representation to be invariant to single and multiple task dialogues using an auxiliary loss. The results from our experiments highlight the difficulty of even the sophisticated variant of transformer model in learning to compose multiple tasks from single task dialogues.

This leads to the desire of an accurate space encoding that enables performance prediction In this paper, we propose an approach to neural architecture via surrogates and black-box optimization to find search (NAS) based on graph embeddings. NAS has high-performing architectures in a continuous search space been addressed previously using discrete, sampling based [67]. Zhang et al. [67] propose D-VAE, a graph neural network methods, which are computationally expensive as well as (GNN) [14, 23, 56] based variational neural architecture differentiable approaches, which come at lower costs but embedding with emphasis on the information flow and enforce stronger constraints on the search space. The proposed thereby achieve good results in architecture performance approach leverages advantages from both sides by prediction and BO on the ENAS search space [39] and on a building a smooth variational neural architecture embedding dataset of Bayesian Networks.

Deep Neural Networks (DNNs) are being deployed in a wide range of settings today, from safety-critical applications like autonomous driving to commercial applications involving image classifications. However, recent research has shown that DNNs can be brittle to even slight variations of the input data. Therefore, rigorous testing of DNNs has gained widespread attention. While DNN robustness under norm-bound perturbation got significant attention over the past few years, our knowledge is still limited when natural variants of the input images come. These natural variants, e.g. a rotated or a rainy version of the original input, are especially concerning as they can occur naturally in the field without any active adversary and may lead to undesirable consequences. Thus, it is important to identify the inputs whose small variations may lead to erroneous DNN behaviors. The very few studies that looked at DNN's robustness under natural variants, however, focus on estimating the overall robustness of DNNs across all the test data rather than localizing such error-producing points. This work aims to bridge this gap. To this end, we study the local per-input robustness properties of the DNNs and leverage those properties to build a white-box (DEEPROBUST-W) and a black-box (DEEPROBUST-B) tool to automatically identify the non-robust points. Our evaluation of these methods on nine DNN models spanning three widely used image classification datasets shows that they are effective in flagging points of poor robustness. In particular, DEEPROBUST-W and DEEPROBUST-B are able to achieve an F1 score of up to 91.4% and 99.1%, respectively. We further show that DEEPROBUST-W can be applied to a regression problem for a self-driving car application.