We combine supervised learning with unsupervised learning in deep neural networks. The proposed model is trained to simultaneously minimize the sum of supervised and unsupervised cost functions by backpropagation, avoiding the need for layer-wise pre-training. Our work builds on top of the Ladder network proposed by Valpola (2015) which we extend by combining the model with supervision. We show that the resulting model reaches state-of-the-art performance in semi-supervised MNIST and CIFAR-10 classification in addition to permutation-invariant MNIST classification with all labels.

Google's Deep Mind software took just 40 days to become the best ever player of the ancient game of Go, and commentators heralded it as a major milestone for deep learning, a field of artificial intelligence (AI). The achievement highlighted how computers equipped with the right algorithms can now quickly teach themselves to achieve a specific goal. This approach, known as unsupervised learning, works well within a well-defined set of parameters, such as a board game. But what about when something unexpected happens, such as a change in the rules of the game? In the work environment, unexpected things happen all the time, limiting the usefulness of today's AI systems.

"It will look like one huge brain from our perspective," he says, "in much the same way that the internet looks like one big thing." That impression will be an illusion, but we will think it all the same. It is only way our limited human brains will be able to comprehend an internet of connected artificial intelligences. Valpola has set himself the task of building this future network. But at the moment his goal seems far away. Despite all the advances made in recent years, the Finnish computer scientist is disappointed with the rate of progress in artificial intelligence.

A software library for constructing and learning probabilistic models is presented. The library offers a set of building blocks from which a large variety of static and dynamic models can be built. These include hierarchical models for variances of other variables and many nonlinear models. The underlying variational Bayesian machinery, providing for fast and robust estimation but being mathematically rather involved, is almost completely hidden from the user thus making it very easy to use the library. The building blocks include Gaussian, rectified Gaussian and mixture-of-Gaussians variables and computational nodes which can be combined rather freely.