The variational autoencoder (VAE) is a popular probabilistic generative model. However, one shortcoming of VAEs is that the latent variables cannot be discrete, which makes it difficult to generate data from different modes of a distribution. Here, we propose an extension of the VAE framework that incorporates a classifier to infer the discrete class of the modeled data. To model sequential data, we can combine our Classifying VAE with a recurrent neural network such as an LSTM. We apply this model to algorithmic music generation, where our model learns to generate musical sequences in different keys. Most previous work in this area avoids modeling key by transposing data into only one or two keys, as opposed to the 10+ different keys in the original music. We show that our Classifying VAE and Classifying VAE+LSTM models outperform the corresponding non-classifying models in generating musical samples that stay in key. This benefit is especially apparent when trained on untransposed music data in the original keys.

With widespread adoption of electronic health records, there is an increased emphasis for predictive models that can effectively deal with clinical time-series data. Powered by Recurrent Neural Network (RNN) architectures with Long Short-Term Memory (LSTM) units, deep neural networks have achieved state-of-the-art results in several clinical prediction tasks. Despite the success of RNNs, its sequential nature prohibits parallelized computing, thus making it inefficient particularly when processing long sequences. Recently, architectures which are based solely on attention mechanisms have shown remarkable success in transduction tasks in NLP, while being computationally superior. In this paper, for the first time, we utilize attention models for clinical time-series modeling, thereby dispensing recurrence entirely. We develop the \textit{SAnD} (Simply Attend and Diagnose) architecture, which employs a masked, self-attention mechanism, and uses positional encoding and dense interpolation strategies for incorporating temporal order. Furthermore, we develop a multi-task variant of \textit{SAnD} to jointly infer models with multiple diagnosis tasks. Using the recent MIMIC-III benchmark datasets, we demonstrate that the proposed approach achieves state-of-the-art performance in all tasks, outperforming LSTM models and classical baselines with hand-engineered features.

Modeling sequential data has become more and more important in practice. Some applications are autonomous driving, virtual sensors and weather forecasting. To model such systems so called recurrent models are used. In this article we introduce two new Deep Recurrent Gaussian Process (DRGP) models based on the Sparse Spectrum Gaussian Process (SSGP) and the improved variational version called Variational Sparse Spectrum Gaussian Process (VSSGP). We follow the recurrent structure given by an existing DRGP based on a specific sparse Nystr\"om approximation. Therefore, we also variationally integrate out the input-space and hence can propagate uncertainty through the layers. We can show that for the resulting lower bound an optimal variational distribution exists. Training is realized through optimizing the variational lower bound. Using Distributed Variational Inference (DVI), we can reduce the computational complexity. We improve over current state of the art methods in prediction accuracy for experimental data-sets used for their evaluation and introduce a new data-set for engine control, named Emission. Furthermore, our method can easily be adapted for unsupervised learning, e.g. the latent variable model and its deep version.

Summary: This is the second in our chatbot series. Here we explore Natural Language Understanding (NLU), the front end of all chatbots. We'll discuss the programming necessary to build rules based chatbots and then look at the use of deep learning algorithms that are the basis for AI enabled chatbots. In our last article which was the first in this series about chatbots we covered the basics including their brief technological history, uses, basic design choices, and where deep learning comes into play. In this installment we'll explore in more depth how Natural Language Understanding (NLU) based on deep neural net RNN/LSTMs enables both rules based and AI chatbots.

When I was beginning my way in data science, I often faced the problem of choosing the most appropriate algorithm for my specific problem. If you're like me, when you open some article about machine learning algorithms, you see dozens of detailed descriptions. The paradox is that they don't ease the choice. In this article for Statsbot, I will try to explain basic concepts and give some intuition of using different kinds of machine learning algorithms in different tasks. At the end of the article, you'll find the structured overview of the main features of described algorithms. Supervised learning Supervised learning is the task of inferring a function from labeled training data.

Summary: Reinforcement Learning (RL) is likely to be the next big push in artificial intelligence. But the concept of modeling in RL is very different from our statistical techniques and deep learning. In this two part series we'll take a look at the basics of RL models, how they're built and used. In the next part, we'll address some of the complexities that make development a challenge. Now that we have pretty much conquered speech, text, and image processing with deep neural nets, it's time to turn our attention to what comes next. It's likely that the next most important area of development for AI will be reinforcement learning (RL).

Industrial NTL detection systems are still largely based on expert knowledge when deciding whether to carry out costly on-site inspections of customers. Electricity providers are reluctant to move to large-scale deployments of machine learning systems as the latter may suggest a large number of unnecessary inspections. Therefore, electricity providers want to understand why a specific customer was predicted to cause electricity theft or not. As a consequence, the models used should be interpretable, for example by using decision tree models rather than black box-like models such as deep learning. We have also recently proposed a method for visualizing prediction results at various granularity levels in a spatial hologram.

N. Jean, M. Burke, M. Xie, W. M. Davis, D. B. Lobell, and S. Ermon, "Combining satellite imagery and machine learning to predict poverty," Science 353, 790–794 (2016). B. Forster, D. Van De Ville, J. Berent, D. Sage, and M. Unser, "Complex wavelets for extended depth-of-field: a new method for the fusion of multichannel microscopy images," Microsc.

Two of the top numerical platforms in Python that provide the basis for Deep Learning research and development are Theano and TensorFlow. Both are very powerful libraries, but both can be difficult to use directly for creating deep learning models. In this post, you will discover the Keras Python library that provides a clean and convenient way to create a range of deep learning models on top of Theano or TensorFlow. Introduction to the Python Deep Learning Library Keras Photo by Dennis Jarvis, some rights reserved. Keras is a minimalist Python library for deep learning that can run on top of Theano or TensorFlow.

Users of R have long been deprived of the opportunity to join the deep learning movement while remaining within the same programming language. With the release of MXNet, the situation began to change, but the frequent updates to the original documentation and changes that break backward compatibility still limit the popularity of this library. TensorFlow, Theano, CNTK) combined with detailed documentation and a lot of examples looks much more attractive. This article presents a solution to the problem of segmenting images in Carvana Image Masking Challenge, in which you want to learn how to separate cars photographed from 16 different angles will be dismantled. The neural network part is fully implemented on Keras, image processing is answered by magick (interface to ImageMagick), and parallel processing is provided by parallel doParallel foreach (Windows) or parallel doMC foreach (Linux).