In this episode of AI defined Jon explains Deep Learning a more advanced form of machine learning.

Our system works in two stages; first we train a transformer model on a very large amount of data in an unsupervised manner -- using language modeling as a training signal -- then we fine-tune this model on much smaller supervised datasets to help it solve specific tasks. We developed this approach following our sentiment neuron work, in which we noted that unsupervised learning techniques can yield surprisingly discriminative features when trained on enough data. Here, we wanted to further explore this idea: can we develop one model, train it in an unsupervised way on a large amount of data, and then fine-tune the model to achieve good performance on many different tasks? Our results indicate that this approach works surprisingly well; the same core model can be fine-tuned for very different tasks with minimal adaptation. This work builds on the approach introduced in Semi-supervised Sequence Learning, which showed how to improve document classification performance by using unsupervised pre-training of an LSTM followed by supervised fine-tuning.

When it comes to deep learning chip startups, hype moves fast but crossing the finish line to real production silicon takes an incredibly long time. There are several incumbents on the custom hardware side aiming for the AI training and inference market but outside of Google's TPU, there are very few functioning inside datacenters. From the forthcoming Nervana chips (now expected in 2019) to startups like Graphcore, Cerebras (which just ducked back into stealth mode), among several others, the pressure is on to create hardware that reflects the latest framework and algorithmic developments that so far seem to run quite well on widely available GPUs with all the requisite porting and software work handled thanks to big library and tooling investments from Nvidia over the last few years. In other words, it is going to be damn tough to beat Nvidia, especially this late in the game, but for one of the better known deep learning chip startups, Wave Computing, there is more going on in the outfield than we might readily see. This is why the company has invested in tech that might seem a bit left field--that is, until we look at how the AI hardware game of the future might play out.

Parallel sentence extraction is a task addressing the data sparsity problem found in multilingual natural language processing applications. We propose a bidirectional recurrent neural network based approach to extract parallel sentences from collections of multilingual texts. Our experiments with noisy parallel corpora show that we can achieve promising results against a competitive baseline by removing the need of specific feature engineering or additional external resources. To justify the utility of our approach, we extract sentence pairs from Wikipedia articles to train machine translation systems and show significant improvements in translation performance.

Resting-state functional MRI (rs-fMRI) scans hold the potential to serve as a diagnostic or prognostic tool for a wide variety of conditions, such as autism, Alzheimer's disease, and stroke. While a growing number of studies have demonstrated the promise of machine learning algorithms for rs-fMRI based clinical or behavioral prediction, most prior models have been limited in their capacity to exploit the richness of the data. For example, classification techniques applied to rs-fMRI often rely on region-based summary statistics and/or linear models. In this work, we propose a novel volumetric Convolutional Neural Network (CNN) framework that takes advantage of the full-resolution 3D spatial structure of rs-fMRI data and fits non-linear predictive models. We showcase our approach on a challenging large-scale dataset (ABIDE, with N > 2,000) and report state-of-the-art accuracy results on rs-fMRI-based discrimination of autism patients and healthy controls.

We introduce Generative Neural Machine Translation (GNMT), a latent variable architecture which is designed to model the semantics of the source and target sentences. We modify an encoder-decoder translation model by adding a latent variable as a language agnostic representation which is encouraged to learn the meaning of the sentence. GNMT achieves competitive BLEU scores on pure translation tasks, and is superior when there are missing words in the source sentence. We augment the model to facilitate multilingual translation and semi-supervised learning without adding parameters. This framework significantly reduces overfitting when there is limited paired data available, and is effective for translating between pairs of languages not seen during training.

Abstract--In a world of global trading, maritime safety, security and efficiency are crucial issues. We propose a multi-task deep learning framework for vessel monitoring using Automatic Identification System (AIS) data streams. We combine recurrent neural networks with latent variable modeling and an embedding of AIS messages to a new representation space to jointly address key issues to be dealt with when considering AIS data streams: massive amount of streaming data, noisy data and irregular timesampling. We demonstrate the relevance of the proposed deep learning framework on real AIS datasets for a three-task setting, namely trajectory reconstruction, anomaly detection and vessel type identification. In the world of a globalized economy, maritime surveillance is a vital demand. Besides, the real-time delivery of maritime situation maps is also necessary for a variety of activities: fishing activities control, smuggling detection, EEZ intrusion detection, transshipment detection, maritime pollution monitoring, etc. Over the last decades, the development of terrestrial networks and satellite constellations of Automatic Identification System (AIS) has opened a new era in maritime traffic surveillance. Every day, AIS provides on a global scale hundreds of millions of messages [1], which contain ships' identifiers, their Global Positioning System (GPS) coordinates, their speed, course, etc. The potential of this massive amount of data is clearly of interest if tools and models provide means to efficiently extract, detect and analyze relevant information from these data streams. However, current operational systems, which strongly rely on human experts, can only deal with a limited fraction of AIS data streams. Thus, the development of AIbased systems is a critical challenge.

In target tracking, the estimation of an unknown weaving target frequency is crucial for improving the miss distance. The estimation process is commonly carried out in a Kalman framework. The objective of this paper is to examine the potential of using neural networks in target tracking applications. To that end, we propose estimating the weaving frequency using deep neural networks, instead of classical Kalman framework based estimation. Particularly, we focus on the case where a set of possible constant target frequencies is known. Several neural network architectures, requiring low computational resources were designed to estimate the unknown frequency out of the known set of frequencies. The proposed approach performance is compared with the multiple model adaptive estimation algorithm. Simulation results show that in the examined scenarios, deep neural network outperforms multiple model adaptive estimation in terms of accuracy and the amount of required measurements to convergence.

We propose a simple and easy to implement neural network compression algorithm that achieves results competitive with more complicated state-of-the-art methods. The key idea is to modify the original optimization problem by adding K independent Gaussian priors (corresponding to the k-means objective) over the network parameters to achieve parameter quantization, as well as an L1 penalty to achieve pruning. Unlike many existing quantization-based methods, our method uses hard clustering assignments of network parameters, which adds minimal change or overhead to standard network training. We also demonstrate experimentally that tying neural network parameters provides less gain in generalization performance than changing network architecture and connectivity patterns entirely.

DNA is perceived as a sequence over the letters {A, C, G, T }, the alphabet of nucleotides. This sequence constitutes the code that acts as a blueprint for all processes taking place in a cell. But beyond merely reflecting primary sequence, DNA is a molecule, which implies that DNA assumes spatial structure and shape. The spatial organization of DNA is achieved by integrating ("recruiting") other molecules, the histone proteins, that help to assume the correct spatial configuration. The combination of DNA and helper molecules is called chromatin; the spatial configuration of the chromatin, finally, defines the functional properties of local areas of the DNA [de Graaf and van Steensel, 2013]. Chromatin can assume several function-defining epigenetic states, where states vary along the genome [Ernst et al., 2011]. The key determinant for spatial configuration is the underlying primary DNA sequence: sequential patterns are responsible for recruiting histone proteins and their chemical modifications, which in turn give rise to or even define the chromatin states. The exact configuration of the chromatin and its interplay with the underlying raw DNA sequence are under active research.