Statistical relational AI (StarAI) aims at reasoning and learning in noisy domains described in terms of objects and relationships by combining probability with first-order logic. With huge advances in deep learning in the current years, combining deep networks with first-order logic has been the focus of several recent studies. Many of the existing attempts, however, only focus on relations and ignore object properties. The attempts that do consider object properties are limited in terms of modelling power or scalability. In this paper, we develop relational neural networks (RelNNs) by adding hidden layers to relational logistic regression (the relational counterpart of logistic regression). We learn latent properties for objects both directly and through general rules. Back-propagation is used for training these models. A modular, layer-wise architecture facilitates utilizing the techniques developed within deep learning community to our architecture. Initial experiments on eight tasks over three real-world datasets show that RelNNs are promising models for relational learning.

Conventional decision trees have a number of favorable properties, including interpretability, a small computational footprint and the ability to learn from little training data. However, they lack a key quality that has helped fuel the deep learning revolution: that of being end-to-end trainable, and to learn from scratch those features that best allow to solve a given supervised learning problem. Recent work (Kontschieder 2015) has addressed this deficit, but at the cost of losing a main attractive trait of decision trees: the fact that each sample is routed along a small subset of tree nodes only. We here propose a model and Expectation-Maximization training scheme for decision trees that are fully probabilistic at train time, but after a deterministic annealing process become deterministic at test time. We also analyze the learned oblique split parameters on image datasets and show that Neural Networks can be trained at each split node. In summary, we present the first end-to-end learning scheme for deterministic decision trees and present results on par with or superior to published standard oblique decision tree algorithms.

Highly distributed training of Deep Neural Networks (DNNs) on future compute platforms (offering 100 of TeraOps/s of computational capacity) is expected to be severely communication constrained. To overcome this limitation, new gradient compression techniques are needed that are computationally friendly, applicable to a wide variety of layers seen in Deep Neural Networks and adaptable to variations in network architectures as well as their hyper-parameters. In this paper we introduce a novel technique - the Adaptive Residual Gradient Compression (AdaComp) scheme. AdaComp is based on localized selection of gradient residues and automatically tunes the compression rate depending on local activity. We show excellent results on a wide spectrum of state of the art Deep Learning models in multiple domains (vision, speech, language), datasets (MNIST, CIFAR10, ImageNet, BN50, Shakespeare), optimizers (SGD with momentum, Adam) and network parameters (number of learners, minibatch-size etc.). Exploiting both sparsity and quantization, we demonstrate end-to-end compression rates of ~200X for fully-connected and recurrent layers, and ~40X for convolutional layers, without any noticeable degradation in model accuracies.

This work proposes a novel solution to the problem of internal covariate shift and dying neurons using the concept of linked neurons. We define the neuron linkage in terms of two constraints: first, all neuron activations in the linkage must have the same operating point. That is to say, all of them share input weights. Secondly, a set of neurons is linked if and only if there is at least one member of the linkage that has a non-zero gradient in regard to the input of the activation function. This means that for any input in the activation function, there is at least one member of the linkage that operates in a non-flat and non-zero area. This simple change has profound implications in the network learning dynamics. In this article we explore the consequences of this proposal and show that by using this kind of units, internal covariate shift is implicitly solved. As a result of this, the use of linked neurons allows to train arbitrarily large networks without any architectural or algorithmic trick, effectively removing the need of using re-normalization schemes such as Batch Normalization, which leads to halving the required training time. It also solves the problem of the need for standarized input data. Results show that the units using the linkage not only do effectively solve the aforementioned problems, but are also a competitive alternative with respect to state-of-the-art with very promising results.

Deep learning (DL), a new-generation artificial neural network research, has made profound strides in recent years. This review paper is intended to provide water resources scientists with a simple technical overview, trans-disciplinary progress update, and potentially inspirations about DL. Effective architectures, more accessible data, advances in regularization, and new computing power enabled the success of DL. A trans-disciplinary review reveals that DL is rapidly transforming myriad scientific disciplines including high-energy physics, astronomy, chemistry, genomics and remote sensing, where systematic DL toolkits, innovative customizations, and sub-disciplines have emerged. However, with a few exceptions, its adoption in hydrology has so far been gradual. The literature suggests that novel regularization techniques can effectively prevent high-capacity deep networks from overfitting. As a result, in most scientific disciplines, DL models demonstrated superior predictive and generalization performance to conventional methods. Meanwhile, less noticed is that DL may also serve as a scientific exploratory tool. A new area termed "AI neuroscience", has been born. This budding sub-discipline is accumulating a significant body of work, e.g., distilling knowledge obtained in DL networks to interpretable models, attributing decisions to inputs via back-propagation of relevance, or visualization of activations. These methods are designed to interpret the decision process of deep networks and derive insights. While scientists so far have mostly been using customized, ad-hoc methods for interpretation, vast opportunities await for DL to propel advancement in water science.

Training deep neural networks requires massive amounts of training data, but for many tasks only limited labeled data is available. This makes weak supervision attractive, using weak or noisy signals like the output of heuristic methods or user click-through data for training. In a semi-supervised setting, we can use a large set of data with weak labels to pretrain a neural network and then fine-tune the parameters with a small amount of data with true labels. This feels intuitively sub-optimal as these two independent stages leave the model unaware about the varying label quality. What if we could somehow inform the model about the label quality? In this paper, we propose a semi-supervised learning method where we train two neural networks in a multi-task fashion: a "target network" and a "confidence network". The target network is optimized to perform a given task and is trained using a large set of unlabeled data that are weakly annotated. We propose to weight the gradient updates to the target network using the scores provided by the second confidence network, which is trained on a small amount of supervised data. Thus we avoid that the weight updates computed from noisy labels harm the quality of the target network model. We evaluate our learning strategy on two different tasks: document ranking and sentiment classification. The results demonstrate that our approach not only enhances the performance compared to the baselines but also speeds up the learning process from weak labels.

Jon Ludvig Hammer, the Norwegian grandmaster, described AlphaZero's strategy as'insane attacking chess' which was coupled with'profound' positional play. The DeepMind team eventually want to use the algorithm to solve big health problems. They believe that the programme could come up with cures for major illness in a matter of days or weeks, which would have taken humans hundreds of years to find. The company has already begun using AlphaZero to study protein folding and has promised it will soon publish new findings. Misfolded proteins are responsible for many devastating diseases, including Alzheimer's, Parkinson's and cystic fibrosis.

A leading provider of data science education, Springboard was just named one of the best data science bootcamps in the world by SwitchUp for the second year in a row! Springboard recently overhauled the course to give students an even better learning experience via their online, mentor-led curriculum. They took feedback from students, alumni, and mentors, and combined it with deep industry research to make their courses better--here's how: The curriculum now includes cutting edge teachings in deep learning and machine learning. Dig deep into artificial intelligence with new course modules, and learn one of today's most in-demand skills. They partnered with leaders at Datacamp--experts in teaching R and Python--to update the rest of the curriculum too.

Google says its AlphaGo Zero artificial intelligence program has triumphed at chess against world-leading specialist software within hours of teaching itself the game from scratch. The firm's DeepMind division says that it played 100 games against Stockfish 8, and won or drew all of them. The research has yet to be peer reviewed. But experts already suggest the achievement will strengthen the firm's position in a competitive sector. "From a scientific point of view, it's the latest in a series of dazzling results that DeepMind has produced," the University of Oxford's Prof Michael Wooldridge told the BBC.

Google's DeepMind team has already advanced its AlphaGo AI to dominate Go without human input, but now the system is clever enough to master other board games without intervention. Researchers have developed a more generalized system for AlphaGo Zero that can train itself to achieve "superhuman" skill in chess, Shogi (a Japanese classic) and other game types knowing only the rules, all within less than a day. It doesn't need example games or other references. This doesn't mean that DeepMind has developed a truly general purpose, independent AI... yet. Chess and Shogi were relatively easy tests, as they're simpler than Go. It'll be another thing entirely to tackle complex video games like StarCraft II, let alone fuzzier concepts like walking or abstract thought.