Developments in deep generative models have allowed for tractable learning of high-dimensional data distributions. While the employed learning procedures typically assume that training data is drawn i.i.d. from the distribution of interest, it may be desirable to model distinct distributions which are observed sequentially, such as when different classes are encountered over time. Although conditional variations of deep generative models permit multiple distributions to be modeled by a single network in a disentangled fashion, they are susceptible to catastrophic forgetting when the distributions are encountered sequentially. In this paper, we adapt recent work in reducing catastrophic forgetting to the task of training generative adversarial networks on a sequence of distinct distributions, enabling continual generative modeling.

Adaptive optimization methods, which perform local optimization with a metric constructed from the history of iterates, are becoming increasingly popular for training deep neural networks. Examples include AdaGrad, RMSProp, and Adam. We show that for simple overparameterized problems, adaptive methods often find drastically different solutions than gradient descent (GD) or stochastic gradient descent (SGD). We construct an illustrative binary classification problem where the data is linearly separable, GD and SGD achieve zero test error, and AdaGrad, Adam, and RMSProp attain test errors arbitrarily close to half. We additionally study the empirical generalization capability of adaptive methods on several state-of-the-art deep learning models. We observe that the solutions found by adaptive methods generalize worse (often significantly worse) than SGD, even when these solutions have better training performance. These results suggest that practitioners should reconsider the use of adaptive methods to train neural networks.

We address the problem using three different architectures that combine fine and coarse features to obtain the final segmentation. We compare three different networks that use multi-resolution features in terms of both design and performance and we show that they improve their single-resolution counterparts.

The generalization error of deep neural networks via their classification margin is studied in this work. Our approach is based on the Jacobian matrix of a deep neural network and can be applied to networks with arbitrary non-linearities and pooling layers, and to networks with different architectures such as feed forward networks and residual networks. Our analysis leads to the conclusion that a bounded spectral norm of the network's Jacobian matrix in the neighbourhood of the training samples is crucial for a deep neural network of arbitrary depth and width to generalize well. This is a significant improvement over the current bounds in the literature, which imply that the generalization error grows with either the width or the depth of the network. Moreover, it shows that the recently proposed batch normalization and weight normalization re-parametrizations enjoy good generalization properties, and leads to a novel network regularizer based on the network's Jacobian matrix. The analysis is supported with experimental results on the MNIST, CIFAR-10, LaRED and ImageNet datasets.

Deep learning libraries and frameworks such as Theano, Keras, Caffe, and TensorFlow have gained enormous recent popularity. In fact, Google's TensorFlow is the most starred machine learning repository on Github. TensorFlow, despite being in the wild for little more than 6 months, has captured such a formidable market share that one could argue that it has become the default deep learning library by a large swath of seasoned neural network veterans and newcomers alike. It's not the only library to consider, obviously. There are many others, a few of which are mentioned above.

Jeremy P. Howard, @JeremyPHoward, is a leading Machine Learning and Deep learning researcher and entrepreneur. His current startup is fast.ai Previously, he was CEO and founder of Enlitic, Kaggle President, and #1 ranked Kaggle competitor. Jeremy initiatives attracts a lot of attention in the industry, so I was very interested to learn from him about his latest project, a first Deep Learning for coders MOOC at course.fast.ai. The course is totally free and includes no advertising - Jeremy created it purely as a service to the community.

A shot of Amazon's automated store Amazon Go Right from Google's predictive search, Apple's Siri to Facebook's automatic friend tagging and Uber using artificial intelligence (AI) applications, AI is making big noise and has got attention of the world in a massive way. While AI is increasingly everywhere what you probably don't hear much about is AI in retail. While it may not sound as sexy as AI applications empowering recommendation engines for Google and Netflix, AI in retail will affect how we all shop, be it online or offline. AI and deep learning uses software programs to perform complex tasks without active participation or insight from humans making the task at hand a hundred times easier and faster to complete. Armed with the ability to research a retail brand online, customers are today more knowledgeable than ever before.

Everyday, we depend on many systems and machines. We use a car to travel, a lift go up and down, and a plane to fly. Electricity comes through turbines and in a hospital machine keeps us alive. Some failures are an just an inconvenience, while others could mean life or death. When stakes are high, we perform regular maintenance on our systems. For example, cars are serviced once every few months and aircrafts are serviced daily.

Several organizations are already using drones to fight poaching, but the Lindbergh Foundation is taking it one step further. The environmental non-profit has joined forces with Neurala in order to use the company's deep learning neural network AI to boost the capabilities of the drones in its Air Shepherd program. Neurala taught its technology what elephants, rhinos and poachers look like, so it can accurately pinpoint and mark them in videos. It will now put the AI to work sifting through all the footage the foundation's drones beam back in real time, including infrared footage taken at night. The AI's job is to pore over these videos and quickly identify the presence of poachers to prevent them from even reaching the animals' herds. It's the perfect addition to the Air Shepherd program that aims to use cutting edge software and drones to stop poaching in Africa.

This article comes from GitHub. A curated list of resources dedicated to bayesian deep learning. To read the original article, click here.