Abstract-- Conventional deep learning classifiers are static in the sense that they are trained on a predefined set of classes and learning to classify a novel class typically requires retraining. In this work, we address the problem of Low-Shot networkexpansion learning. We introduce a learning framework which enables expanding a pre-trained (base) deep network to classify novel classes when the number of examples for the novel classes is particularly small. We present a simple yet powerful hard distillation method where the base network is augmented with additional weights to classify the novel classes, while keeping the weights of the base network unchanged. We show that since only a small number of weights needs to be trained, the hard distillation excels in low-shot training scenarios. Furthermore, hard distillation avoids detriment to classification performance on the base classes. Finally, we show that low-shot network expansion can be done with a very small memory footprint by using a compact generative model of the base classes training data with only a negligible degradation relative to learning with the full training set. I. INTRODUCTION In many real-life scenarios, a fast and simple classifier expansion is required to extend the set of classes that a deep network can classify.

We consider adversarial online learning in a non-convex setting under the assumption that the learner has an access to an offline optimization oracle. In the most general unstructured setting of prediction with expert advice, Hazan and Koren (2015) established an exponential gap demonstrating that online learning can be significantly harder. Interestingly, this gap is eliminated once we assume a convex structure. A natural question which arises is whether the convexity assumption can be dropped. In this work we answer this question in the affirmative. Namely, we show that online learning is computationally equivalent to statistical learning in the Lipschitz-bounded setting. Notably, most deep neural networks satisfy these assumptions. We prove this result by adapting the ubiquitous Follow-The-Perturbed-Leader paradigm of Kalai and Vempala (2004). As an application we demonstrate how the offline oracle enables efficient computation of an equilibrium in non-convex games, that include GAN (generative adversarial networks) as a special case.

Current state-of-the-art reading comprehension models rely heavily on recurrent neural networks. We explored an entirely different approach to question answering: a convolutional model. By their nature, these convolutional models are fast to train and capture local dependencies well, though they can struggle with longer-range dependencies and thus require augmentation to achieve comparable performance to RNN-based models. We conducted over two dozen controlled experiments with convolutional models and various kernel/attention/regularization schemes to determine the precise performance gains of each strategy, while maintaining a focus on speed. We ultimately ensembled three models: crossconv (0.5398 dev F1), attnconv (0.5665), and maybeconv (0.5285). The ensembled model was able to achieve a 0.6238 F1 score using the official SQuAD evaluation script. Our individual convolutional model crossconv was able to exceed the performance of the RNN-plus-attention baseline by 25% while training 6 times faster.

This work views neural networks as data generating systems and applies anomalous pattern detection techniques on that data in order to detect when a network is processing an anomalous input. Detecting anomalies is a critical component for multiple machine learning problems including detecting adversarial noise. More broadly, this work is a step towards giving neural networks the ability to recognize an out-of-distribution sample. This is the first work to introduce "Subset Scanning" methods from the anomalous pattern detection domain to the task of detecting anomalous input of neural networks. Subset scanning treats the detection problem as a search for the most anomalous subset of node activations (i.e., highest scoring subset according to non-parametric scan statistics). Mathematical properties of these scoring functions allow the search to be completed in log-linear rather than exponential time while still guaranteeing the most anomalous subset of nodes in the network is identified for a given input. Quantitative results for detecting and characterizing adversarial noise are provided for CIFAR-10 images on a simple convolutional neural network. We observe an "interference" pattern where anomalous activations in shallow layers suppress the activation structure of the original image in deeper layers.

And more broadly, why should you use JVM languagues like Java, Scala, Clojure or Kotlin to build AI and machine-learning solutions? Java is the most widely used programming language in the world. Large organizations in the public and private sector have enormous Java code bases, and rely heavily on the JVM as a compute environment. In particular, much of the open-source big data stack is written for the JVM. This includes Apache Hadoop for distributed data management; Apache Spark as a distributed run-time for fast ETL; Apache Kafka as a message queue; ElasticSearch, Apache Lucene and Apache Solr for search; and Apache Cassandra for data storage to name a few.

As world rolling around Artificial Intelligence (AI), demand for the AI-based product seen exponential growth, so the AI research. Deep learning algorithms and techniques are widely used for research and development of these products. Good news is that year by year Deep Learning has seen its glory in the release of many open source frameworks which ease the pain to develop and implement these algorithms. As there are many deep learning frameworks out there and it can lead to confusion as to which one is better for your task. And choosing a deep learning framework for an AI project is as important as choosing a programming language to code product, Data science project coupled with the right deep learning framework has truly amplified the overall productivity.

Graph Nets is DeepMind's library for building graph networks in Tensorflow and Sonnet. A graph network takes a graph as input and returns a graph as output. The input graph has edge- (E), node- (V), and global-level (u) attributes. The output graph has the same structure, but updated attributes. Graph networks are part of the broader family of "graph neural networks" (Scarselli et al., 2009).

Holding certain layers frozen on a network and training is effectively the same as training on a transformed version of the input, the transformed version being the intermediate outputs at the boundary of the frozen layers. This is the process of "feature extraction" from the input data and will be referred to as "featurizing" in this document. The forward pass to "featurize" the input data on large, pertained networks can be time consuming. DL4J also provides a TransferLearningHelper class with the following capabilities. When running multiple epochs users will save on computation time since the expensive forward pass on the frozen layers/vertices will only have to be conducted once.

Last week we wrapped up a highly successful GPU Technology Conference (GTC) Europe in Munich! GTC is NVIDIA's international conference series, bringing together the top minds in deep learning, analytics, and of course GPUs for sessions, workshops, keynotes, and more. This was the place to be for any and all European organizations interested in leveraging the power of the GPU. As the Kinetica engine runs on GPUs, there's no better place for us to share our solutions for advanced analytics and deep learning. This year we noticed a significant increase in the number of organizations that understand the challenges of the Extreme Data Economy.

The Hong Kong Polytechnic University (PolyU) recently developed an intelligent fabric defect detection system, called "WiseEye," which leverages advanced technologies including artificial Intelligence (AI) and Deep Learning in the process of quality control (QC) in textile industry. It helps to save manpower and to enhance the automation management in the textile manufacturing. Supported by AI-based machine-vision technology, "WiseEye" can be installed in a weaving machine to help fabric manufacturers to detect defects instantly in the production process. Through the automatic inspection system, the production line manager can detect the defects, thus helping them to identify the cause of the problems and fix them immediately. "WiseEye" is developed by the Textile and Apparel Artificial Intelligence (TAAI) Research Team, which is spearheaded by Professor Calvin Wong, Cheng Yik Hung Professor in Fashion of Institute of Textiles and Clothing, PolyU.