Deployed Deep Neural Networks (DNNs) are often trained further to improve in performance. This complicates the tracking of DNN model versions and the synchronization between already deployed models and upstream updates of the same architecture. Software Version Control cannot be applied straight-forwardly to DNNs due to the different nature of software and DNN models. In this paper we investigate if the weight space of DNN models contains a structure, which can be used for the identification of individual DNN models. Our results show that DNN models evolve on unique, smooth trajectories in weight space which we can exploit as feature for DNN version control.

In recent years, increasingly large datasets with two different sets of features measured for each sample have become prevalent in many areas of biology. For example, a recently developed method called Patch-seq provides single-cell RNA sequencing data together with electrophysiological measurements of the same neurons. However, the efficient and interpretable analysis of such paired data has remained a challenge. As a tool for exploration and visualization of Patch-seq data, we introduce neural networks with a two-dimensional bottleneck, trained to predict electrophysiological measurements from gene expression. To make the model biologically interpretable and perform gene selection, we enforce sparsity by using a group lasso penalty, followed by pruning of the input units and subsequent fine-tuning. We applied this method to a recent dataset with $>$1000 neurons from mouse motor cortex and found that the resulting bottleneck model had the same predictive performance as a full-rank linear model with much higher latent dimensionality. Exploring the two-dimensional latent space in terms of neural types showed that the nonlinear bottleneck approach led to much better visualizations and higher biological interpretability.

We introduce DREAM, a deep reinforcement learning algorithm that finds optimal strategies in imperfect-information games with multiple agents. Formally, DREAM converges to a Nash Equilibrium in two-player zero-sum games and to an extensive-form coarse correlated equilibrium in all other games. Our primary innovation is an effective algorithm that, in contrast to other regret-based deep learning algorithms, does not require access to a perfect simulator of the game to achieve good performance. We show that DREAM empirically achieves state-of-the-art performance among model-free algorithms in popular benchmark games, and is even competitive with algorithms that do use a perfect simulator.

Solving long-tail large vocabulary object detection with deep learning based models is a challenging and demanding task, which is however under-explored.In this work, we provide the first systematic analysis on the underperformance of state-of-the-art models in front of long-tail distribution. We find existing detection methods are unable to model few-shot classes when the dataset is extremely skewed, which can result in classifier imbalance in terms of parameter magnitude. Directly adapting long-tail classification models to detection frameworks can not solve this problem due to the intrinsic difference between detection and classification.In this work, we propose a novel balanced group softmax (BAGS) module for balancing the classifiers within the detection frameworks through group-wise training. It implicitly modulates the training process for the head and tail classes and ensures they are both sufficiently trained, without requiring any extra sampling for the instances from the tail classes.Extensive experiments on the very recent long-tail large vocabulary object recognition benchmark LVIS show that our proposed BAGS significantly improves the performance of detectors with various backbones and frameworks on both object detection and instance segmentation. It beats all state-of-the-art methods transferred from long-tail image classification and establishes new state-of-the-art.Code is available at https://github.com/FishYuLi/BalancedGroupSoftmax.

In this paper, we consider matrix completion with absolute deviation loss and obtain an estimator of the median matrix. Despite several appealing properties of median, the non-smooth absolute deviation loss leads to computational challenge for large-scale data sets which are increasingly common among matrix completion problems. A simple solution to large-scale problems is parallel computing. However, embarrassingly parallel fashion often leads to inefficient estimators. Based on the idea of pseudo data, we propose a novel refinement step, which turns such inefficient estimators into a rate (near-)optimal matrix completion procedure. The refined estimator is an approximation of a regularized least median estimator, and therefore not an ordinary regularized empirical risk estimator. This leads to a non-standard analysis of asymptotic behaviors. Empirical results are also provided to confirm the effectiveness of the proposed method.

Market Basket Analysis (MBA) is a popular technique to identify associations between products, which is crucial for business decision making. Previous studies typically adopt conventional frequent itemset mining algorithms to perform MBA. However, they generally fail to uncover rarely occurring associations among the products at their most granular level. Also, they have limited ability to capture temporal dynamics in associations between products. Hence, we propose OMBA, a novel representation learning technique for Online Market Basket Analysis. OMBA jointly learns representations for products and users such that they preserve the temporal dynamics of product-to-product and user-to-product associations. Subsequently, OMBA proposes a scalable yet effective online method to generate products' associations using their representations. Our extensive experiments on three real-world datasets show that OMBA outperforms state-of-the-art methods by as much as 21%, while emphasizing rarely occurring strong associations and effectively capturing temporal changes in associations.

Kernel methods provide an elegant and principled approach to nonparametric learning, but so far could hardly be used in large scale problems, since na\"ive implementations scale poorly with data size. Recent advances have shown the benefits of a number of algorithmic ideas, for example combining optimization, numerical linear algebra and random projections. Here, we push these efforts further to develop and test a solver that takes full advantage of GPU hardware. Towards this end, we designed a preconditioned gradient solver for kernel methods exploiting both GPU acceleration and parallelization with multiple GPUs, implementing out-of-core variants of common linear algebra operations to guarantee optimal hardware utilization. Further, we optimize the numerical precision of different operations and maximize efficiency of matrix-vector multiplications. As a result we can experimentally show dramatic speedups on datasets with billions of points, while still guaranteeing state of the art performance. Additionally, we make our software available as an easy to use library.

Generative Adversarial Network(GAN) provides a good generative framework to produce realistic samples, but suffers from two recognized issues as mode collapse and unstable training. In this work, we propose to employ explicit manifold learning as prior to alleviate mode collapse and stabilize training of GAN. Since the basic assumption of conventional manifold learning fails in case of sparse and uneven data distribution, we introduce a new target, Minimum Manifold Coding (MMC), for manifold learning to encourage simple and unfolded manifold. In essence, MMC is the general case of the shortest Hamiltonian Path problem and pursues manifold with minimum Riemann volume. Using the standardized code from MMC as prior, GAN is guaranteed to recover a simple and unfolded manifold covering all the training data. Our experiments on both the toy data and real datasets show the effectiveness of MMCGAN in alleviating mode collapse, stabilizing training, and improving the quality of generated samples.

Due to their high computational efficiency on a continuous space, gradient optimization methods have shown great potential in the neural architecture search (NAS) domain. The mapping of network representation from the discrete space to a latent space is the key to discovering novel architectures, however, existing gradient-based methods fail to fully characterize the networks. In this paper, we propose an efficient NAS approach to optimize network architectures in a continuous space, where the latent space is built upon variational autoencoder (VAE) and graph neural networks (GNN). The framework jointly learns four components: the encoder, the performance predictor, the complexity predictor and the decoder in an end-to-end manner. The encoder and the decoder belong to a graph VAE, mapping architectures between continuous representations and network architectures. The predictors are two regression models, fitting the performance and computational complexity, respectively. Those predictors ensure the discovered architectures characterize both excellent performance and high computational efficiency. Extensive experiments demonstrate our framework not only generates appropriate continuous representations but also discovers powerful neural architectures.

Learning the distribution of observed data is a basic task in unsupervised learning which has been studied for decades. The recently-introduced concept of Generative Adversarial Networks (GANs) [1] has demonstrated great success in various distribution learning tasks. Unlike the traditional maximum-likelihood-based approaches, GANs learn the distribution of observed data through a zero-sum game between two machine players, a generator G mimicking the true distribution of data and a discriminator D distinguishing the generator's produced samples from real data points. This zero-sum game is typically formulated through a minimax optimization problem where G and D optimize a minimax objective quantifying how dissimilar G's generated samples and real training samples are. In GAN minimax optimization problems, the generator and discriminator functions are commonly chosen as two deep neural networks (DNNs). Leveraging the expressive power of DNNs, GANs have achieved state-of-the-art performance in learning complex distributions of image data [2, 3, 4]. This success, however, is achieved at the cost of their notoriously difficult training procedure which has introduced several challenges to the machine learning community. Addressing these challenges requires a deeper theoretical understanding of GANs, including their approximation, generalization, and optimization properties. Specifically, GANs have been frequently observed to fail in learning multi-modal distributions [5].