We examine the role of memorization in deep learning, drawing connections to capacity, generalization, and adversarial robustness. While deep networks are capable of memorizing noise data, our results suggest that they tend to prioritize learning simple patterns first. In our experiments, we expose qualitative differences in gradient-based optimization of deep neural networks (DNNs) on noise vs. real data. We also demonstrate that for appropriately tuned explicit regularization (e.g., dropout) we can degrade DNN training performance on noise datasets without compromising generalization on real data. Our analysis suggests that the notions of effective capacity which are dataset independent are unlikely to explain the generalization performance of deep networks when trained with gradient based methods because training data itself plays an important role in determining the degree of memorization.

Before I started my most recent job at ThinkTopic, the concepts of "functional programming" and "machine learning" belonged to two different worlds entirely. One was a programming paradigm surging in popularity as the world turned towards simplicity, composability, and immutability to maintain complex scaling applications; the other was a tool to teach computers to autocomplete doodles and make music. The more I worked with the two, the more I began realizing that the overlap is both practical and theoretical. Firstly, machine learning is not a stand-alone endeavor; it needs to be rapidly incorporated into complex scaling applications in industry. Secondly, machine learning -- and deep learning in particular -- is functional by design.

Continuing our series of deep learning updates, we pulled together some of the awesome resources that have emerged since our last post. In case you missed it, here are our past updates: June (part 1, part 2, part 3), May, April (part 1, part 2), March part 1, February, November, September part 2 & October part 1, September part 1, August (part 1, part 2), July (part 1, part 2), June, and the original set of 20 resources we outlined in April 2016. As always, this list is not comprehensive, so let us know if there's something we should add, or if you're interested in discussing this area further. Teaching an AI agent to learn & apply language. Here we present an agent that learns to interpret language in a simulated 3D environment where it is rewarded for the successful execution of written instructions.

We're open-sourcing a high-performance Python library for robotic simulation using the MuJoCo engine, developed over our past year of robotics research. This library is one of our core tools for deep learning robotics research, which we've now released as a major version of mujoco-py, our Python 3 bindings for MuJoCo. Many methods in trajectory optimization and reinforcement learning (like LQR, PI2, and TRPO) benefit from being able to run multiple simulations in parallel. Naive usage of the new version's MjSimPool interface shows a 400% speedup over the old, and still about 180% over an optimized and restricted usage pattern using Python's multiprocessing package to gain the same level of parallelism. The majority of the speedup comes from reduced access times to the various MuJoCo data structures.

The idea came about from Microsoft Labs teams in Redmond and Bangalore, India. By doing that, they were able to make an image detection system run about 20 times faster on a Raspberry Pi 3 without any loss of accuracy. "There is just no way to take a deep neural network, have it stay as accurate as it is today, and consume 10,000 times less resources. To get some new ideas and help, they've made some of their early training tools and algorithms available to Raspberry Pi hobbyists and other researchers on Github.

Imagine the basic magical things that you do everyday that you never think highly of. You just don't think too much as to why can you'see' things, 'hear' things, 'say' things, 'understand' things, whereas things can't do the same. Computers started off as objects that made our work a bit easier by taking the calculation work on their shoulders. Now we're just trying to put more of our work on its head because, why not? Its shoulders are already busy with our basic calculation work but there's space on its head too! But quite ironically, we're working really hard to make that happen.

When you're far from a cell tower and need to figure out if that bluebird is Sialia sialis or Sialia mexicana, no cloud server is going to help you. That's why companies are squeezing AI onto portable devices, and Microsoft has just taken that to a new extreme by putting deep learning algorithms onto a Raspberry Pi. The goals is to get AI onto "dumb" devices like sprinklers, medical implants and soil sensors to make them more useful, even if there's no supercomputer or internet connection in sight. The idea came about from Microsoft Labs teams in Redmond and Bangalore, India. Ofer Dekel, who manages an AI optimization group at the Redmond Lab, was trying to figure out a way to stop squirrels from eating flower bulbs and seeds from his bird feeder.

In general there is no guarantee that, even with a lot of data, deep learning does better than other techniques, for example tree-based such as random forest or boosted trees. Do I need some Skynet to run it?

The rise of artificial intelligence in recent years is grounded in the success of deep learning. Three major drivers caused the breakthrough of (deep) neural networks: the availability of huge amounts of training data, powerful computational infrastructure, and advances in academia. Thereby deep learning systems start to outperform not only classical methods, but also human benchmarks in various tasks like image classification or face recognition. This creates the potential for many disruptive new businesses leveraging deep learning to solve real-world problems. At Berlin-based Merantix, we work on these new business cases in various industries (currently automotive, health, financial and advertising).