Deep learning 2016 the year in review

In order to understand trends in the field, I find it helpful to think of developments in deep learning as being driven by three major frontiers that limit the success of artificial intelligence in general and deep learning in particular. Firstly, there is the available computing power and infrastructure, such as fast GPUs, cloud services providers (have you checked out Amazon's new EC2 P2 instance?) and tools (Tensorflow, Torch, Keras etc), secondly, there is the amount and quality of the training data and thirdly, the algorithms (CNN, LSTM, SGD) using the training data and running on the hardware. Invariably behind every new development or advancement, lies an expansion of one of these frontiers. Much of the progress we have seen this year is driven by an expansion of the former two frontiers; we now have systems that are able to recognize images and speech with an accuracy that rivals that of humans and there is an abundance of data and tools to develop them. However, almost all of these systems rely on supervised learning and thereby on the ready availability of labelled data sets.