Over the past year, AI has seriously improved its ability to generate the written word. By scanning huge datasets of text, machine learning software can produce convincing samples of everything from short stories to song lyrics. Now, those same techniques are being applied to the world of coding with a new program called Deep TabNine. Deep TabNine is what's known as a coding autocompleter. Programmers can install it as an add-on in their editor of choice, and when they start writing, it'll suggest how to continue each line, offering small chunks at a time.

As language learning makes a sea change to online, users will come to expect personalized learning experiences. China's AI-powered online education market alone reached $568 million in 2017 and is expected to surpass $26 billion in 2022. At Sana Labs, we build AI technologies to power these learning experiences through easy to integrate APIs. This means that machine learning models for personalization as well as pronunciation, grammar, and overall fluency feedback can be production ready in days, not months. In this article, I'll highlight why deep learning will power this shift.

James Fisher recently wrote about a simple "inception bar" phishing attack he created for mobile users of Chrome: scrolling down in the Chrome app on a mobile device hides the URL bar to maximize screen space. Fisher demonstrated how an attacker can simply insert an image of a faux Chrome URL bar with whatever domain they're trying to impersonate, and--with a little CSS and JavaScript magic--they can fool the user into thinking they're on a legitimate site like hsbc.com instead of SomePhishingSite.com. As Fisher points out in his proof-of-concept blog post, there are various user-interface changes Google could make to prevent or at least mitigate this phishing technique--but as a thought experiment, let's imagine some ways machine learning (ML) could be leveraged to detect and prevent this sort of attack. One effective countermeasure may be to use computer vision techniques to classify individual images. In particular, when dealing with images, the most obvious modeling approaches entail using convolutional neural networks.

We introduce the HSIC (Hilbert-Schmidt independence criterion) bottleneck for training deep neural networks. The HSIC bottleneck is an alternative to conventional backpropagation, that has a number of distinct advantages. The method facilitates parallel processing and requires significantly less operations. It does not suffer from exploding or vanishing gradients. It is biologically more plausible than backpropagation as there is no requirement for symmetric feedback. We find that the HSIC bottleneck provides a performance on the MNIST/FashionMNIST/CIFAR10 classification comparable to backpropagation with a cross-entropy target, even when the system is not encouraged to make the output resemble the classification labels. Appending a single layer trained with SGD (without backpropagation) results in state-of-the-art performance.

We present the first unsupervised deep learning method for pollen analysis using bright-field microscopy. Using a modest dataset of 650 images of pollen grains collected from honey, we achieve family level identification of pollen. We embed images of pollen grains into a low-dimensional latent space and compare Euclidean and Riemannian metrics on these spaces for clustering. We propose this system for automated analysis of pollen and other microscopic biological structures which have only small or unlabelled datasets available.

Deep Neural Networks (DNNs) deliver state-of-the-art performance in many image recognition and understanding applications. However, despite their outstanding performance, these models are black-boxes and it is hard to understand how they make their decisions. Over the past few years, researchers have studied the problem of providing explanations of why DNNs predicted their results. However, existing techniques are either obtrusive, requiring changes in model training, or suffer from low output quality. In this paper, we present a novel method, NeuroMask, for generating an interpretable explanation of classification model results. When applied to image classification models, NeuroMask identifies the image parts that are most important to classifier results by applying a mask that hides/reveals different parts of the image, before feeding it back into the model. The mask values are tuned by minimizing a properly designed cost function that preserves the classification result and encourages producing an interpretable mask. Experiments using state-of-the-art Convolutional Neural Networks for image recognition on different datasets (CIFAR-10 and ImageNet) show that NeuroMask successfully localizes the parts of the input image which are most relevant to the DNN decision. By showing a visual quality comparison between NeuroMask explanations and those of other methods, we find NeuroMask to be both accurate and interpretable.

In this paper, we theoretically prove that gradient descent can find a global minimum for nonlinear deep neural networks of sizes commonly encountered in practice. The theory developed in this paper requires only the number of trainable parameters to increase linearly as the number of training samples increases. This allows the size of the deep neural networks to be several orders of magnitude smaller than that required by the previous theories. Moreover, we prove that the linear increase of the size of the network is the optimal rate and that it cannot be improved, except by a logarithmic factor. Furthermore, deep neural networks with the trainability guarantee are shown to generalize well to unseen test samples with a natural dataset but not a random dataset.

--Malaria is a life-threatening disease affecting millions. Microscopy-based assessment of thin blood films is a standard method to (i) determine malaria species and (ii) quanti-tate high-parasitemia infections. Full automation of malaria microscopy by machine learning (ML) is a challenging task because field-prepared slides vary widely in quality and presentation, and artifacts often heavily outnumber relatively rare parasites. In this work, we describe a complete, fully-automated framework for thin film malaria analysis that applies ML methods, including convolutional neural nets (CNNs), trained on a large and diverse dataset of field-prepared thin blood films. Quanti-tation and species identification results are close to sufficiently accurate for the concrete needs of drug resistance monitoring and clinical use-cases on field-prepared samples. We focus our methods and our performance metrics on the field use-case requirements. We discuss key issues and important metrics for the application of ML methods to malaria microscopy. Index T erms --malaria, automated microscopy, deep neural networks, gradient boosted trees I. I NTRODUCTION Malaria is a mosquito-borne disease caused by Plasmodium species ( P . Manual microscopy examination of Giemsa-stained blood films is a widespread malaria diagnosis method. Key use-cases include diagnosis; species identification (ID) to guide treatment [2]; and quantitation of parasites for drug resistance studies, to track how fast a drug clears parasites from the blood. However, a lack of training, high inter-sample variability in preparation and presentation, and difficult field conditions can result in poor accuracy [3], [4]. Also, lack of trained personnel limits the number of drug resistance sentinel sites. Malaria microscopy is a difficult task for automated image-processing and machine learning (ML) systems for two reasons: Field-prepared blood films vary widely in quality and presentation; and parasites are small (with feature size close to optical limits of resolution), rare, highly variable, and easily confused with non-parasite objects (artifacts). But it is also a high-value target, due to the potential benefit for so many people, and also because automated systems have some concrete advantages: They can be widely deployed, solving the expert-training bottleneck; they can examine more blood volume per patient, reducing variability in quantitation caused by Poisson statistics; and their results are reproducible.

Under StepDecay learning rate strategy (decaying the learning rate after pre-defined epochs), it is a common phenomenon that the trajectories of learning statistics (training loss, test loss, test accuracy, etc.) are divided into several stages by sharp transitions. This paper studies the phenomenon in detail. Carefully designed experiments suggest the root cause to be the stochasticity of SGD. The convincing fact is the phenomenon disappears when Batch Gradient Descend is adopted. We then propose a hypothesis about the mechanism behind the phenomenon: the noise from SGD can be magnified to several levels by different learning rates, and only certain patterns are learnable within a certain level of noise. Patterns that can be learned under large noise are called easy patterns and patterns only learnable under small noise are called complex patterns. We derive several implications inspired by the hypothesis: (1) Since some patterns are not learnable until the next stage, we can design an algorithm to automatically detect the end of the current stage and switch to the next stage to expedite the training. The algorithm we design (called AutoDecay) shortens the time for training ResNet50 on ImageNet by $ 10 $\% without hurting the performance. (2) Since patterns are learned with increasing complexity, it is possible they have decreasing transferability. We study the transferability of models learned in different stages. Although later stage models have superior performance on ImageNet, we do find that they are less transferable. The verification of these two implications supports the hypothesis about the mechanism.

Deep neural networks have revolutionized many real world applications, due to their flexibility in data fitting and accurate predictions for unseen data. A line of research reveals that neural networks can approximate certain classes of functions with an arbitrary accuracy, while the size of the network scales exponentially with respect to the data dimension. Empirical results, however, suggest that networks of moderate size already yield appealing performance. To explain such a gap, a common belief is that many data sets exhibit low dimensional structures, and can be modeled as samples near a low dimensional manifold. In this paper, we prove that neural networks can efficiently approximate functions supported on low dimensional manifolds. The network size scales exponentially in the approximation error, with an exponent depending on the intrinsic dimension of the data and the smoothness of the function. Our result shows that exploiting low dimensional data structures can greatly enhance the efficiency in function approximation by neural networks. We also implement a sub-network that assigns input data to their corresponding local neighborhoods, which may be of independent interest.