Network pruning is aimed at imposing sparsity in a neural network architecture by increasing the portion of zero-valued weights for reducing its size regarding energy-efficiency consideration and increasing evaluation speed. In most of the conducted research efforts, the sparsity is enforced for network pruning without any attention to the internal network characteristics such as unbalanced outputs of the neurons or more specifically the distribution of the weights and outputs of the neurons. That may cause severe accuracy drop due to uncontrolled sparsity. In this work, we propose an attention mechanism that simultaneously controls the sparsity intensity and supervised network pruning by keeping important information bottlenecks of the network to be active. On CIFAR-10, the proposed method outperforms the best baseline method by 6% and reduced the accuracy drop by 2.6x at the same level of sparsity.

Effective training of neural networks requires much data. In the low-data regime, parameters are underdetermined, and learnt networks generalise poorly. Data Augmentation alleviates this by using existing data more effectively. However standard data augmentation produces only limited plausible alternative data. Given there is potential to generate a much broader set of augmentations, we design and train a generative model to do data augmentation. The model, based on image conditional Generative Adversarial Networks, takes data from a source domain and learns to take any data item and generalise it to generate other within-class data items. As this generative process does not depend on the classes themselves, it can be applied to novel unseen classes of data. We show that a Data Augmentation Generative Adversarial Network (DAGAN) augments standard vanilla classifiers well. We also show a DAGAN can enhance few-shot learning systems such as Matching Networks. We demonstrate these approaches on Omniglot, on EMNIST having learnt the DAGAN on Omniglot, and VGG-Face data. In our experiments we can see over 13% increase in accuracy in the low-data regime experiments in Omniglot (from 69% to 82%), EMNIST (73.9% to 76%) and VGG-Face (4.5% to 12%); in Matching Networks for Omniglot we observe an increase of 0.5% (from 96.9% to 97.4%) and an increase of 1.8% in EMNIST (from 59.5% to 61.3%).

We interpret part of the experimental results of Shwartz-Ziv and Tishby [2017]. Inspired by these results, we established a conjecture of the dynamics of the machinary of deep neural network. This conjecture can be used to explain the counterpart result by Saxe et al. [2018].

Knowledge Graph Embedding methods aim at representing entities and relations in a knowledge base as points or vectors in a continuous vector space. Several approaches using embeddings have shown promising results on tasks such as link prediction, entity recommendation, question answering, and triplet classification. However, only a few methods can compute low-dimensional embeddings of very large knowledge bases. In this paper, we propose KG2Vec, a novel approach to Knowledge Graph Embedding based on the skip-gram model. Instead of using a predefined scoring function, we learn it relying on Long Short-Term Memories. We evaluated the goodness of our embeddings on knowledge graph completion and show that KG2Vec is comparable to the quality of the scalable state-of-the-art approaches and can process large graphs by parsing more than a hundred million triples in less than 6 hours on common hardware.

Decades of research on the neural code underlying spatial navigation have revealed a diverse set of neural response properties. The Entorhinal Cortex (EC) of the mammalian brain contains a rich set of spatial correlates, including grid cells which encode space using tessellating patterns. However, the mechanisms and functional significance of these spatial representations remain largely mysterious. As a new way to understand these neural representations, we trained recurrent neural networks (RNNs) to perform navigation tasks in 2D arenas based on velocity inputs. Surprisingly, we find that grid-like spatial response patterns emerge in trained networks, along with units that exhibit other spatial correlates, including border cells and band-like cells. All these different functional types of neurons have been observed experimentally. The order of the emergence of grid-like and border cells is also consistent with observations from developmental studies. Together, our results suggest that grid cells, border cells and others as observed in EC may be a natural solution for representing space efficiently given the predominant recurrent connections in the neural circuits.

Where some businesses are employing artificial intelligence to sell you more, IBM is using it to sell you less. Specifically, it's employing one set of AI tools to minimize the amount of compute time on its cloud services you need to buy in order to train another set of AI tools to run your business. That will also allow IBM's customers to make the most of another scarce and expensive resource, AI expertise, according to Ruchir Puri, Chief Architect for IBM Watson and an IBM Fellow. "We're lowering the barrier to entry for machine learning capabilities for enterprise," Puri said. The barrier Puri is talking of is the scarcity of human expertise in deep learning, a way of training an artificial intelligence in a particular domain of expertise.

Say hello to the capsule network. AI has enjoyed huge growth in the past few years, and much of that success is owed to deep neural networks, which provide the smarts behind impressive tricks like image recognition. But there is growing concern that some of the fundamental principles that have made those systems so successful may not be able to overcome the major problems facing AI--perhaps the biggest of which is a need for huge quantities of data from which to learn (for a deep dive on this, check out our feature "Is AI Riding a One-Trick Pony?"). Google's Geoff Hinton appears to be among those fretting about AI's future. As Wired reports, Hinton has unveiled a new take on traditional neural networks that he calls capsule networks.

The next Windows 10 update opens the way for the integration of artificial intelligence within Windows applications, directly impacting hundreds of millions of devices from Windows PCs and tablets to IoT Edge devices. The new version of the Windows ML platform allows developers to integrate pre-trained deep-learning models within their applications directly in Visual Studio. The models must be converted into the Open Neural Network Exchange (ONNX) format before importing into VS tools. ONNX is an open-source machine-learning framework launched by Microsoft and Facebook in September 2017, later joined by AWS. ONNX enables portability between neural-network frameworks, making it possible for models trained with tools like Pytorch, Apache MxNet, caffe2 or Microsoft Cognitive Toolkit (CNTK) to be translated to ONNX and later implemented in Windows applications.

A public report by Harvard reveals how unprepared the US Military is when it comes to the Artificial Intelligence (AI) technology known as Deep Learning. The study by Harvard's Kennedy center was published in July 2017, written by Greg Allen Taniel Chan, and was conducted with funding from IARPA. The research is titled "Artificial Intelligence and National Security". I've written about the many tribes of AI and about the use of the term AI being too ambiguous and meaning too many things to too many people. Where do we find Deep Learning in this report from Harvard?

In 2017, artificial intelligence attracted $12 billion of VC investment. We are only beginning to discover the usefulness of AI applications. Amazon recently unveiled a brick-and-mortar grocery store that has successfully supplanted cashiers and checkout lines with computer vision, sensors, and deep learning. Between the investment, the press coverage, and the dramatic innovation, "AI" has become a hot buzzword. But does it even exist yet?