Graph convolutional neural networks have demonstrated promising performance in attributed graph learning, thanks to the use of graph convolution that effectively combines graph structures and node features for learning node representations. However, one intrinsic limitation of the commonly adopted 1-D graph convolution is that it only exploits graph connectivity for feature smoothing, which may lead to inferior performance on sparse and noisy real-world attributed networks. To address this problem, we propose to explore relational information among node attributes to complement node relations for representation learning. In particular, we propose to use 2-D graph convolution to jointly model the two kinds of relations and develop a computationally efficient dimensionwise separable 2-D graph convolution (DSGC). Theoretically, we show that DSGC can reduce intra-class variance of node features on both the node dimension and the attribute dimension to facilitate learning. Empirically, we demonstrate that by incorporating attribute relations, DSGC achieves significant performance gain over state-of-the-art methods on node classification and clustering on several real-world attributed networks.

In this work, we introduce a deep learning-based polar code construction algorithm. The core idea is to represent the information/frozen bit indices of a polar code as a binary vector which can be interpreted as trainable weights of a neural network (NN). For this, we demonstrate how this binary vector can be relaxed to a soft-valued vector, facilitating the learning process through gradient descent and enabling an efficient code construction. We further show how different polar code design constraints (e.g., code rate) can be taken into account by means of careful binary-to-soft and soft-to-binary conversions, along with rate-adjustment after each learning iteration. Besides its conceptual simplicity, this approach benefits from having the "decoder-in-the-loop", i.e., the nature of the decoder is inherently taken into consideration while learning (designing) the polar code. We show results for belief propagation (BP) decoding over both AWGN and Rayleigh fading channels with considerable performance gains over state-of-the-art construction schemes.

(Frankle & Carbin, 2019) shows that there exist winning tickets (small but critical subnetworks) for dense, randomly initialized networks, that can be trained alone to achieve comparable accuracies to the latter in a similar number of iterations. However, the identification of these winning tickets still requires the costly train-prune-retrain process, limiting their practical benefits. In this paper, we discover for the first time that the winning tickets can be identified at the very early training stage, which we term as early-bird (EB) tickets, via low-cost training schemes (e.g., early stopping and low-precision training) at large learning rates. Our finding of EB tickets is consistent with recently reported observations that the key connectivity patterns of neural networks emerge early. Furthermore, we propose a mask distance metric that can be used to identify EB tickets with low computational overhead, without needing to know the true winning tickets that emerge after the full training. Finally, we leverage the existence of EB tickets and the proposed mask distance to develop efficient training methods, which are achieved by first identifying EB tickets via low-cost schemes, and then continuing to train merely the EB tickets towards the target accuracy. Experiments based on various deep networks and datasets validate: 1) the existence of EB tickets, and the effectiveness of mask distance in efficiently identifying them; and 2) that the proposed efficient training via EB tickets can achieve up to 4.7x energy savings while maintaining comparable or even better accuracy, demonstrating a promising and easily adopted method for tackling cost-prohibitive deep network training.

In hyperspectral image (HSI) classification, spatial context has demonstrated its significance in achieving promising performance. However, conventional spatial context-based methods simply assume that spatially neighboring pixels should correspond to the same land-cover class, so they often fail to correctly discover the contextual relations among pixels in complex situations, and thus leading to imperfect classification results on some irregular or inhomogeneous regions such as class boundaries. To address this deficiency, we develop a new HSI classification method based on the recently proposed Graph Convolutional Network (GCN), as it can flexibly encode the relations among arbitrarily structured non-Euclidean data. Different from traditional GCN, there are two novel strategies adopted by our method to further exploit the contextual relations for accurate HSI classification. First, since the receptive field of traditional GCN is often limited to fairly small neighborhood, we proposed to capture long range contextual relations in HSI by performing successive graph convolutions on a learned region-induced graph which is transformed from the original 2D image grids. Second, we refine the graph edge weight and the connective relationships among image regions by learning the improved adjacency matrix and the 'edge filter', so that the graph can be gradually refined to adapt to the representations generated by each graph convolutional layer. Such updated graph will in turn result in accurate region representations, and vice versa. The experiments carried out on three real-world benchmark datasets demonstrate that the proposed method yields significant improvement in the classification performance when compared with some state-of-the-art approaches.

ABSTRACT V oice activity detection (V AD), which classifies frames as speech or non-speech, is an important module in many speech applications including speaker verification. In this paper, we propose a novel method, called self-adaptive soft V AD, to incorporate a deep neural network (DNN)-based V AD into a deep speaker embedding system. The proposed method is a combination of the following two approaches. The first approach is soft V AD, which performs a soft selection of frame-level features extracted from a speaker feature extractor. The frame-level features are weighted by their corresponding speech posteriors estimated from the DNN-based V AD, and then aggregated to generate a speaker embedding. The second approach is self-adaptive V AD, which fine-tunes the pre-trained V AD on the speaker verification data to reduce the domain mismatch. Here, we introduce two unsupervised domain adaptation (DA) schemes, namely speech posterior-based DA (SP-DA) and joint learning-based DA (JL-DA). Experiments on a Korean speech database demonstrate that the verification performance is improved significantly in real-world environments by using self-adaptive soft V AD. Index T erms-- speaker verification, voice activity detection, unsupervised domain adaptation, soft V AD 1. INTRODUCTION Speaker verification (SV) is the task of verifying a person's claimed identity based on his or her voice.

One of the biggest issues in deep learning theory is the gener alization ability of networks with huge model size. The classical learning the ory suggests that overparameterized models cause overfitting. However, prac tically used large deep models avoid overfitting, which is not well explained by the c lassical approaches. To resolve this issue, several attempts have been made. Amon g them, the compression based bound is one of the promising approaches. However, the compression based bound can be applied only to a compressed network, and i t is not applicable to the non-compressed original network. In this paper, we gi ve a unified framework that can convert compression based bounds to those for n on-compressed original networks. The bound gives even better rate than the one for the compressed network by improving the bias term. By establishing the unified framework, we can obtain a data dependent generalization error bo und which gives a tighter evaluation than the data independent ones.

Abstract--Malware imposes tremendous threats to computer users nowadays. Since signature-based malware detection methods are neither effective nor efficient to identify new malware, many machine learning-based methods have been proposed. A common disadvantage of existing machine learning methods is that they are not based on understanding the full semantic meaning of assembly code of an executable. They rather use short assembly code fragments, because assembly code is usually too long to be modelled in its entirety . Another disadvantage is that those methods have either inferior performance or bad interpretability . T o overcome these challenges, we propose an Interpretable MAware Detector ( I-MAD), which achieves state-of-the-art performance on static malware detection with excellent interpretability . It integrates a hierarchical T ransformer network that can understand assembly code at the basic block, function, and executable level. It also integrates our novel interpretable feed-forward neural network to provide interpretations for its detection results by pointing out the impact of each feature with respect to the prediction. Experiment results show that our model significantly outperforms previous state-of-the-art static malware detection models and presents meaningful interpretations. Since the Internet has become an integral part of people's life, the large volume of malware spreading on it imposes tremendous threats to billions of netizens. Recognizing mal-ware samples downloaded by legitimate users in a timely manner is thus of crucial importance for their protection. Signature-based malware detection methods are widely used in antivirus products [1]. With the signatures extracted by malware analysts, known malware samples or some of their variants can be precisely recognized. However, with obfuscation techniques or even a change of compiler, it is easy to create variants of known malware that perform the same attack, but with literally different executable code. As a result, the previously crafted signatures can no longer recognize them [2]. Furthermore, signature-based detection is also ineffective to detect new and unseen malware in most cases.

Data Sanity Check for Deep Learning Systems via Learnt Assertions Haochuan Lu †, Huanlin Xu †, Nana Liu †, Y angfan Zhou †, Xin Wang † School of Computer Science, Fudan University, Shanghai, China † Shanghai Key Laboratory of Intelligent Information Processing, Shanghai, China Abstract --Deep learning (DL) techniques have demonstrated satisfactory performance in many tasks, even in safety-critical applications. Reliability is hence a critical consideration to DLbased systems. However, the statistical nature of DL makes it quite vulnerable to invalid inputs, i.e., those cases that are not considered in the training phase of a DL model. This paper proposes to perform data sanity check to identify invalid inputs, so as to enhance the reliability of DLbased systems. T o this end, we design and implement a tool to detect behavior deviation of a DL model when processing an input case, and considers it the symptom of invalid input cases. Via a light, automatic instrumentation to the target DL model, this tool extracts the data flow footprints and conducts an assertion-based validation mechanism. The assertions are built automatically, which are specifically-tailored for DL model data flow analysis. Our experiments conducted with real-world scenarios demonstrate that such an assertion-based data sanity check mechanism is effective in identifying invalid input cases. Moreover, SaneDL is lightweight, easy-to-construct, and non-intrusive to the target DL model. I NTRODUCTION In recent years, deep learning (DL) techniques have shown great effectiveness in various aspects. A huge amount of DLbased applications and systems have been proposed in favor of peoples daily life and industrial production [1]-[3], even in safety-critical applications. Image recognition module for auto-driving vehicles [1], for instance, determines what operation should be taken according to the real-time images captured by cameras. In such safety-critical scenarios, any unreliable system misbehavior may cause severe incidents. Reliability is hence of great significance for practical DLbased systems. It is widely-accepted that every software system has its valid input domain [4]-[7]. Inputs staying in such a domain, namely, valid inputs, are expected to receive proper execution results. Unfortunately, in real circumstances, there is no guarantee the inputs are always valid. Anomalous, unexpected inputs may arrive and result in unpredictable misbehavior, which in turn degrades reliability.

Recently, a lot of papers proposed to use neural networks to approximately solve partial differential equations (PDEs). Yet, there has been a lack of flexible framework for convenient experimentation. In an attempt to fill the gap, we introduce a PyDEns-module open-sourced on GitHub. Coupled with capabilities of BatchFlow, open-source framework for convenient and reproducible deep learning, PyDEns-module allows to 1) solve partial differential equations from a large family, including heat equation and wave equation 2) easily search for the best neural-network architecture among the zoo, that includes ResNet and DenseNet 3) fully control the process of model-training by testing different point-sampling schemes. With that in mind, our main contribution goes as follows: implementation of a ready-to-use and open-source numerical solver of PDEs of a novel format, based on neural networks.

A BSTRACT Current deep neural networks can achieve remarkable performance on a single task. However, when the deep neural network is continually trained on a sequence of tasks, it seems to gradually forget the previous learned knowledge. This phenomenon is referred to as catastrophic forgetting and motivates the field called lifelong learning. The central question in lifelong learning is how to enable deep neural networks to maintain performance on old tasks while learning a new task. In this paper, we introduce a novel and effective lifelong learning algorithm, called MixEd stochastic GrAdient (MEGA), which allows deep neural networks to acquire the ability of retaining performance on old tasks while learning new tasks. Extensive experimental results show that the proposed MEGA algorithm significantly advances the state-of-the-art on all four commonly used lifelong learning benchmarks, reducing the error by up to 18%. 1 I NTRODUCTION A significant step towards artificial general intelligence (AGI) is to enable the learning agent to acquire the ability of remembering past experiences while being trained on a continuum of tasks. Current deep neural networks are capable of achieving remarkable performance on a single task (Goodfellow et al., 2016). However when the network is retrained on a new task, its performance drops drastically on previously trained tasks, a phenomenon which is referred to as catastrophic forgetting (Ratcliff, 1990; Robins, 1995; French, 1999; Kirkpatrick et al., 2017).