The Friday afternoon news dump, a grand tradition observed by politicians and capitalists alike, is usually supposed to hide bad news. So it was a little weird that Elon Musk, founder of electric car maker Tesla, and Sam Altman, president of famed tech incubator Y Combinator, unveiled their new artificial intelligence company at the tail end of a weeklong AI conference in Montreal this past December. But there was a reason they revealed OpenAI at that late hour. It wasn't that no one was looking. It was that everyone was looking. When some of Silicon Valley's most powerful companies caught wind of the project, they began offering tremendous amounts of money to OpenAI's freshly assembled cadre of artificial intelligence researchers, intent on keeping these big thinkers for themselves. The last-minute offers--some made at the conference itself--were large enough to force Musk and Altman to delay the announcement of the new startup.

Artificial Intelligence, Deep Learning, Can It Take Over? Bill Gates, Stephen Hawking and Elon Musk first warned us about Artificial Intelligence (AI). Elon Musk then turned around and with other technologists put 1B into starting a nonprofit research effort – OpenAI just to "keep an eye on it"! Facebook, Google, Amazon, Nvidia, Shopify and others are charging full steam at AI and even open sourcing it! So what is all the AI ruckus about? AI has been subject matter for science fiction for a long time now. Every SciFi show you can think of has a intelligent computer or robot as a sidekick or with some prominent role.

Elon Musk has not been shy about his concerns over artificial intelligence turning evil. So it wasn't a surprise in December when Musk announced the formation of OpenAI, an open-source, non-profit focused on advancing "digital intelligence in the way that is most likely to benefit humanity as a whole." That's all well and good, but not much has been revealed about what exactly OpenAI is working on. OpenAI's co-founder and CTO Greg Brockman told Tech Insider that OpenAI is primarily focusing on advancing machine learning, which is the technology that enables computers to learn how to complete tasks through experience. Specifically, the company is focusing on two key types of machine learning that every major tech company is investing in right now.

Elon Musk has not been shy about his concerns over artificial intelligence turning evil. So it wasn't a surprise in December when Musk announced the formation of OpenAI, an open-source, non-profit focused on advancing "digital intelligence in the way that is most likely to benefit humanity as a whole." That's all well and good, but not much has been revealed about what exactly OpenAI is working on. OpenAI's co-founder and CTO told Tech Insider that OpenAI is primarily focusing on advancing machine learning, which is the technology that enables computers to learn how to complete tasks through experience. Specifically, the company is focusing on two key types of machine learning that every major tech company is investing in right now.

As a former research scientist at Google, Ian Goodfellow has had a direct hand in some of the more complex, promising frameworks set to power the future of deep learning in coming years. He spent his first years at the search giant chipping away at TensorFlow, creating new capabilities, including the creation of a new element to the deep learning stack, called generative adversarial networks. And as part of the Google Brain team, he furthered this work and continued to optimize machine learning algorithms used by Google and now, the wider world. Goodfellow has since moved on to the non-profit OpenAI company, where he is further refining what might be possible with generative adversarial networks. The mission of OpenAI is to develop open source tools to further many of the application areas that were showcased this week at the the GPU Technology Conference this week in San Jose, where the emphasis was placed squarely on the future of deep learning, and of course, the role that Nvidia's accelerators will play in the training and execution of neural networks and other machine learning. There has been a fair bit about VR and gaming, of course, but for a company that is placing its best on where the big money for its graphics chips will be in the next decade, the focus is likely not misplaced.

There's a wide class of generative models for which variational methods are the only known practical way to do inference. This includes basically any model with black-box ("neural") dependence relations, and many others as well, e.g., Bayesian nonparametrics for any significant dataset size. The point of variational methods is not to calculate partition functions (although you do get that as a side effect); the point is to fit sophisticated models that have complex latent structure. Which does yield improvements across pretty much any metric you'd care about.

In machine learning, there is an ongoing revival of the use of variational Bayes (VB) to deal with complex probabilistic models with hidden variables. The revival is driven by the use of stochastic methods to approximate the VB lower bound and associated gradients. See for example [1, 2, 5]. The advantages include automated inference [3] and also that they are applicable to a much wider class of probabilistic models. While the basic recipes are limited in the flexibility of the approximate hidden-variable posteriors, there are ongoing efforts to make them more flexible.

Deep generative models (DGMs) are effective on learning multilayered representations of complex data and performing inference of input data by exploring the generative ability. However, little work has been done on examining or empowering the discriminative ability of DGMs on making accurate predictions. This paper presents max-margin deep generative models (mmDGMs), which explore the strongly discriminative principle of max-margin learning to improve the discriminative power of DGMs, while retaining the generative capability. We develop an efficient doubly stochastic subgradient algorithm for the piecewise linear objective. Empirical results on MNIST and SVHN datasets demonstrate that (1) max-margin learning can significantly improve the prediction performance of DGMs and meanwhile retain the generative ability; and (2) mmDGMs are competitive to the state-of-the-art fully discriminative networks by employing deep convolutional neural networks (CNNs) as both recognition and generative models.

A recurring problem when building probabilistic latent variable models is regularization and model selection, for instance, the choice of the dimensionality of the latent space. In the context of belief networks with latent variables, this problem has been adressed with Automatic Relevance Determination (ARD) employing Monte Carlo inference. We present a variational inference approach to ARD for Deep Generative Models using doubly stochastic variational inference to provide fast and scalable learning. We show empirical results on a standard dataset illustrating the effects of contracting the latent space automatically. We show that the resulting latent representations are significantly more compact without loss of expressive power of the learned models.

Sum Product Networks (SPNs) are a recently developed class of deep generative models which compute their associated unnormalized density functions using a special type of arithmetic circuit. When certain sufficient conditions, called the decomposability and completeness conditions (or "D&C" conditions), are imposed on the structure of these circuits, marginal densities and other useful quantities, which are typically intractable for other deep generative models, can be computed by what amounts to a single evaluation of the network (which is a property known as "validity"). However, the effect that the D&C conditions have on the capabilities of D&C SPNs is not well understood. In this work we analyze the D&C conditions, expose the various connections that D&C SPNs have with multilinear arithmetic circuits, and consider the question of how well they can capture various distributions as a function of their size and depth. Among our various contributions is a result which establishes the existence of a relatively simple distribution with fully tractable marginal densities which cannot be efficiently captured by D&C SPNs of any depth, but which can be efficiently captured by various other deep generative models. We also show that with each additional layer of depth permitted, the set of distributions which can be efficiently captured by D&C SPNs grows in size. This kind of "depth hierarchy" property has been widely conjectured to hold for various deep models, but has never been proven for any of them. Some of our other contributions include a new characterization of the D&C conditions as sufficient and necessary ones for a slightly strengthened notion of validity, and various state-machine characterizations of the types of computations that can be performed efficiently by D&C SPNs.