At its core, GPT-2 is a powerful prediction engine. It learned to grasp the structure of the English language by looking at billions of examples of words, sentences, and paragraphs, scraped from the corners of the internet. With that structure, it could then manipulate words into new sentences by statistically predicting the order in which they should appear. So researchers at OpenAI decided to swap the words for pixels and train the same algorithm on images in ImageNet, the most popular image bank for deep learning. Because the algorithm was designed to work with one-dimensional data (i.e., strings of text), they unfurled the images into a single sequence of pixels.

"Artificial Intelligence Project To Help Bring The Power Of The Sun To Earth Is Picked For First U.S. Exascale System" OpenAI's'dangerous' AI text generator is out: People find GPT-2's words'convincing'

In unsupervised learning, variational auto-encoders (VAEs) are an influential class of deep generative models with rich representational power of neural networks and Bayesian methods. However, VAEs suffer from assigning higher likelihood to out-of-distribution (OOD) inputs than in-distribution (ID) inputs. Recent studies advise that the deep generative models with reliable uncertainty estimation is critical to a deep understanding of OOD inputs. Meanwhile, noise contrastive prior (NCP) is an emerging promising method for obtaining uncertainty, with the advantages of easy to scale, being trainable, and compatibility with extensive models. Inspired by these ideas, We propose an improved noise contrastive prior (INCP) to acquire reliable uncertainty estimate for standard VAEs. By combining INCP with the encoder of VAE, patterns between OOD and ID inputs can be well captured and distinguished. Our method outperforms standard VAEs on the FashionMNIST and CIFAR10 datasets. We also demonstrate the preferred robustness of our model by the extensive experiments on anomaly detection tasks.

Machine learning applied to generate data-driven models are lacking of transparency leading the process engineer to lose confidence in relying on the model predictions to optimize his industrial process. Bringing processes in the industry to a certain level of autonomy using data-driven models is particularly challenging as the first user of those models, is the expert in the process with often decades of experience. It is necessary to expose to the process engineer, not solely the model predictions, but also their interpretability. To that end, several approaches have been proposed in the literature. The Local Interpretable Model-agnostic Explanations (LIME) method has gained a lot of interest from the research community recently. The principle of this method is to train a linear model that is locally approximating the black-box model, by generating randomly artificial data points locally. Model-agnostic local interpretability solutions based on LIME have recently emerged to improve the original method. We present in this paper a novel approach, VAE-LIME, for local interpretability of data-driven models forecasting the temperature of the hot metal produced by a blast furnace. Such ironmaking process data is characterized by multivariate time series with high inter-correlation representing the underlying process in a blast furnace. Our contribution is to use a Variational Autoencoder (VAE) to learn the complex blast furnace process characteristics from the data. The VAE is aiming at generating optimal artificial samples to train a local interpretable model better representing the black-box model in the neighborhood of the input sample processed by the black-box model to make a prediction. In comparison with LIME, VAE-LIME is showing a significantly improved local fidelity of the local interpretable linear model with the black-box model resulting in robust model interpretability.

We find that, just as a large transformer model trained on language can generate coherent text, the same exact model trained on pixel sequences can generate coherent image completions and samples. By establishing a correlation between sample quality and image classification accuracy, we show that our best generative model also contains features competitive with top convolutional nets in the unsupervised setting. Unsupervised and self-supervised learning, or learning without human-labeled data, is a longstanding challenge of machine learning. Recently, it has seen incredible success in language, as transformer models like BERT, GPT-2, RoBERTa, T5, and other variants have achieved top performance on a wide array of language tasks. However, the same broad class of models has not been successful in producing strong features for image classification.

We're living in the age of the next industrial revolution: the very first three freed most of the humans from hard labor. This one is aiming to take us over the last domain of human dominance on this planet: our intelligence. In this article, we will put aside ethical, political and social effects of such revolution and concentrate a bit more on the technical side of it. What we see in media today looks a bit different from the real dominance of machines over humans… or not? The most rapidly growing areas of artificial intelligence in the few last years have been computer vision, natural language processing, speech processing and, of course, different customer analytics applications like recommender systems (you may not like it, but targeted advertisements are accurate enough to grow companies' revenues).

Lately, I've been thinking and reading a lot about consciousness and how the human mind works. A question that emerges all the time is whether machines can emulate human thought. An even more interesting one is whether consciousness (a subjective experience) can arise from a machine, but I'll leave that discussion for a future post (I'll need 20 more years to think about that before I can write about it). So, how far are we from _behaviorally _imitating a human? Truth is, we achieved a lot in the past 5 years (see AlphaGo, OpenGPT-2, OpenAI Jukebox, Tesla Autopilot, Alphastar, OpenAI Dota2 Team, OpenAI API), but we're still quite not there.

This lossless compressor achieves a much higher compression rate on English texts than general purpose compressors. Its typical compression ratio is 15% (number of output bits divided by the number of input bits). The compression is achieved by using the probability of the next word computed by the GPT-2 language model released by OpenAI. It is a neural network of 345 million parameters based on the Transformer architecture (the largest GPT-2 model of 1.5 billion parameters brings marginal improvement when compressing short messages). An arithmetic coder generates the bit stream.

Recently, a team of researchers from UC Berkeley and Adobe Research proposed a new machine learning model known as Swapping Autoencoder, which has the capability to perform image manipulation. The key idea of this research is to encode a picture into 2 independent components and then enforce that any swapped combination maps to a realistic image. Deep generative models such as GANs or Generative Adversarial Networks and Variational Autoencoders (VAEs) have gained much traction by the researchers over the years. According to the researchers, deep generative models have become a popular technique when it comes to producing realistic images from randomly sampled data. However, such deep generative models face various challenges when used for a controllable manipulation of existing images.

Jeff Clune is the former Loy and Edith Harris Associate Professor in Computer Science at the University of Wyoming, a Senior Research Manager and founding member of Uber AI Labs, and currently a Research Team Leader at OpenAI. Jeff focuses on robotics and training neural networks via deep learning and deep reinforcement learning. He has also researched open questions in evolutionary biology using computational models of evolution, including studying the evolutionary origins of modularity, hierarchy, and evolvability. Prior to becoming a professor, he was a Research Scientist at Cornell University, received a PhD in computer science and an MA in philosophy from Michigan State University, and received a BA in philosophy from the University of Michigan. More about Jeff's research can be found at JeffClune.com