We give a rigorous analysis of the statistical behavior of gradients in randomly initialized feed-forward networks with ReLU activations. Our results show that a fully connected depth $d$ ReLU net with hidden layer widths $n_j$ will have exploding and vanishing gradients if and only if $\sum_{j=1}^{d-1} 1/n_j$ is large. The point of view of this article is that whether a given neural net will have exploding/vanishing gradients is a function mainly of the architecture of the net, and hence can be tested at initialization. Our results imply that a fully connected network that produces manageable gradients at initialization must have many hidden layers that are about as wide as the network is deep. This work is related to the mean field theory approach to random neural nets. From this point of view, we give a rigorous computation of the $1/n_j$ corrections to the propagation of gradients at the so-called edge of chaos.

This is achieved by creating progressively better representations of sensory inputs, and these representations finally become easily-decoded without any reward or supervision signals [1-3]. This kind of learning is called unsupervised learning, which has long been thought of as a fundamental function of the sensory cortex [4]. Based on the similar computational principle, many layers of artificial neural networks were designed to perform a nonlinear dimensionality reduction of high dimensional data [5], which later triggered resurgence of deep neural networks. By stacking unsupervised modules on top of each other, one can produce a deep feature hierarchy, in which high-level features can be constructed from less abstract ones along the hierarchy. However, it remains rarely explored how this kind of effective representation is transformed along stages of processing.

Deep neural networks (DNNs) have transformed several artificial intelligence research areas including computer vision, speech recognition, and natural language processing. However, recent studies demonstrated that DNNs are vulnerable to adversarial manipulations at testing time. Specifically, suppose we have a testing example, whose label can be correctly predicted by a DNN classifier. An attacker can add a small carefully crafted noise to the testing example such that the DNN classifier predicts an incorrect label, where the crafted testing example is called adversarial example. Such attacks are called evasion attacks. Evasion attacks are one of the biggest challenges for deploying DNNs in safety and security critical applications such as self-driving cars. In this work, we develop new methods to defend against evasion attacks. Our key observation is that adversarial examples are close to the classification boundary. Therefore, we propose region-based classification to be robust to adversarial examples. For a benign/adversarial testing example, we ensemble information in a hypercube centered at the example to predict its label. In contrast, traditional classifiers are point-based classification, i.e., given a testing example, the classifier predicts its label based on the testing example alone. Our evaluation results on MNIST and CIFAR-10 datasets demonstrate that our region-based classification can significantly mitigate evasion attacks without sacrificing classification accuracy on benign examples. Specifically, our region-based classification achieves the same classification accuracy on testing benign examples as point-based classification, but our region-based classification is significantly more robust than point-based classification to various evasion attacks.

Abstract--We apply convolutional neural networks (ConvNets) to the task of distinguishing pathological from normal EEG recordings in the Temple University Hospital EEG Abnormal Corpus. We use two basic, shallow and deep ConvNet architectures recently shown to decode task-related information from EEG at least as well as established algorithms designed for this purpose. In decoding EEG pathology, both ConvNets reached substantially better accuracies (about 6% better, 85% vs. 79%) than the only published result for this dataset, and were still better when using only 1 minute of each recording for training and only six seconds of each recording for testing. We used automated methods to optimize architectural hyperparameters and found intriguingly different ConvNet architectures, e.g., with max pooling as the only nonlinearity. Visualizations of the ConvNet decoding behavior showed that they used spectral power changes in the delta (0-4 Hz) and theta (4-8 Hz) frequency range, possibly alongside other features, consistent with expectations derived from spectral analysis of the EEG data and from the textual medical reports. Analysis of the textual medical reports also highlighted the potential for accuracy increases by integrating contextual information, such as the age of subjects. In summary, the ConvNets and visualization techniques used in this study constitute a next step towards clinically useful automated EEG diagnosis and establish a new baseline for future work on this topic.

Autoregressive models are among the best performing neural density estimators. We describe an approach for increasing the flexibility of an autoregressive model, based on modelling the random numbers that the model uses internally when generating data. By constructing a stack of autoregressive models, each modelling the random numbers of the next model in the stack, we obtain a type of normalizing flow suitable for density estimation, which we call Masked Autoregressive Flow. This type of flow is closely related to Inverse Autoregressive Flow and is a generalization of Real NVP. Masked Autoregressive Flow achieves state-of-the-art performance in a range of general-purpose density estimation tasks.

Although CNNs have been used as early as the nineties to solve character recognition tasks (Le Cunet al., 1997), their current widespread application is due to much more recent work, when a deep CNN was used to beat state-of-the-art in the ImageNet image classification challenge (Krizhevsky et al., 2012). Convolutional neural networks therefore constitute a very useful tool for machine learning practitioners. However, learning to use CNNs for the first time is generally an intimidating experience. A convolutional layer's output shape is affected by the shape of its input as well as the choice of kernel shape, zero padding and strides, and the relationship between these properties is not trivial to infer. This contrasts with fully-connected layers, whose output size is independent of the input size. Additionally, CNNs also usually feature apool-ing stage, adding yet another level of complexity with respect to fully-connected networks. Finally, so-called transposed convolutional layers (also known as fractionally strided convolutional layers) have been employed in more and more work as of late (Zeileret al., 2011; Zeiler and Fergus, 2014; Longet al., 2015; Rad-ford et al., 2015; Visinet al., 2015; Imet al., 2016), and their relationship with convolutional layers has been explained with various degrees of clarity. This guide's objective is twofold: 1. Explain the relationship between convolutional layers and transposed con-volutional layers.

Pulling together deep learning workshops for a large number of students, however, can be a time consuming, error prone, and costly exercise. Furthermore, technical issues with the environment setup and compatibility problems during the workshops impede learning and cause student dissatisfaction. These workshops typically have participants bring their laptops and have them download and install new software. However, with the wide range of laptop platforms (Windows, Mac, Linux), numerous configurations, and version conflicts with existing software, workshops can become frustrating both for presenters and attendees. The RAM and disk space available on laptops and their lack of GPUs affect the types of hands-on labs that can be offered, as deep learning workshops benefit heavily from specialized hardware such as GPUs. An alternative is to build new cloud based custom VMs specifically for the training – this avoids compatibility issues but is quite time consuming and often not reusable based on our experience.

Even Elon Musk and Mark Zuckerberg sparked a heated, and ongoing, debate about whether or not AI will take control of humanity. Indeed, AI continues to demonstrate impressive capabilities. Earlier this year, the AI built into Google's DeepMind AlphaGo defeated the world's top Go player, Chinese professional, Ke Jie. Go is a popular Asian board game reputed to be much more complex than chess. AI is not just for gaming, it also demonstrates a high reliability for detecting cancerous tissue in medical images.

Google is betting big on artificial intelligence as the driver of next generation cloud technology. Microsoft and Amazon have been paying attention. After all, Microsoft and Amazon together dominate the cloud market -- for the moment. To ensure that they remain in the driving seat, they came together last October to offer enterprises an open source development environment called Gluon to carry them forward into the next generation of cloud computing. Gluon is a new deep learning library that allows developers of all skills level to prototype, build, train and deploy machine learning models for cloud, "devices at the edge" and mobile apps.

What are some of the popular optimization algorithms used for training neural networks? This article attempts to answer these questions using a Convolutional Neural Network (CNN) as an example trained on MNIST dataset with TensorFlow. The neural network is represented by f(x(i); theta) where x(i) are the training data and y(i) are the training labels, the gradient of the loss L is computed with respect to model parameters theta. The learning rate (eps_k) determines the size of the step that the algorithm takes along the gradient (in the negative direction in the case of minimization and in the positive direction in the case of maximization). The learning rate is a function of iteration k and is a single most important hyper-parameter.