In this work, we present a globalized stochastic semismooth Newton method for solving stochastic optimization problems involving smooth nonconvex and nonsmooth convex terms in the objective function. We assume that only noisy gradient and Hessian information of the smooth part of the objective function is available via calling stochastic first and second order oracles. The proposed method can be seen as a hybrid approach combining stochastic semismooth Newton steps and stochastic proximal gradient steps. Two inexact growth conditions are incorporated to monitor the convergence and the acceptance of the semismooth Newton steps and it is shown that the algorithm converges globally to stationary points in expectation. Moreover, under standard assumptions and utilizing random matrix concentration inequalities, we prove that the proposed approach locally turns into a pure stochastic semismooth Newton method and converges r-superlinearly with high probability. We present numerical results and comparisons on $\ell_1$-regularized logistic regression and nonconvex binary classification that demonstrate the efficiency of our algorithm.

We propose an approach to address two issues that commonly occur during training of unsupervised GANs. First, since GANs use only a continuous latent distribution to embed multiple classes or clusters of data, they often do not correctly handle the structural discontinuity between disparate classes in a latent space. Second, discriminators of GANs easily forget about past generated samples by generators, incurring instability during adversarial training. We argue that these two infamous problems of unsupervised GAN training can be largely alleviated by a learnable memory network to which both generators and discriminators can access. Generators can effectively learn representation of training samples to understand underlying cluster distributions of data, which ease the structure discontinuity problem. At the same time, discriminators can better memorize clusters of previously generated samples, which mitigate the forgetting problem. We propose a novel end-to-end GAN model named memoryGAN, which involves a memory network that is unsupervisedly trainable and integrable to many existing GAN models. With evaluations on multiple datasets such as Fashion-MNIST, CelebA, CIFAR10, and Chairs, we show that our model is probabilistically interpretable, and generates realistic image samples of high visual fidelity. The memoryGAN also achieves the state-of-the-art inception scores over unsupervised GAN models on the CIFAR10 dataset, without any optimization tricks and weaker divergences.

The macroeconomic climate influences operations with regard to, e.g., raw material prices, financing, supply chain utilization and demand quotas. In order to adapt to the economic environment, decision-makers across the public and private sectors require accurate forecasts of the economic outlook. Existing predictive frameworks base their forecasts primarily on time series analysis, as well as the judgments of experts. As a consequence, current approaches are often biased and prone to error. In order to reduce forecast errors, this paper presents an innovative methodology that extends lag variables with unstructured data in the form of financial news: (1) we apply a variety of models from machine learning to word counts as a high-dimensional input. However, this approach suffers from low interpretability and overfitting, motivating the following remedies. (2) We follow the intuition that the economic climate is driven by general sentiments and suggest a projection of words onto latent semantic structures as a means of feature engineering. (3) We propose a semantic path model, together with estimation technique based on regularization, in order to yield full interpretability of the forecasts. We demonstrate the predictive performance of our approach by utilizing 80,813 ad hoc announcements in order to make long-term forecasts of up to 24 months ahead regarding key macroeconomic indicators. Back-testing reveals a considerable reduction in forecast errors.

Abstraction and realization are bilateral processes that are key in deriving intelligence and creativity. In many domains, the two processes are approached through rules: high-level principles that reveal invariances within similar yet diverse examples. Under a probabilistic setting for discrete input spaces, we focus on the rule realization problem which generates input sample distributions that follow the given rules. More ambitiously, we go beyond a mechanical realization that takes whatever is given, but instead ask for proactively selecting reasonable rules to realize. This goal is demanding in practice, since the initial rule set may not always be consistent and thus intelligent compromises are needed. We formulate both rule realization and selection as two strongly connected components within a single and symmetric bi-convex problem, and derive an efficient algorithm that works at large scale. Taking music compositional rules as the main example throughout the paper, we demonstrate our model's efficiency in not only music realization (composition) but also music interpretation and understanding (analysis).

We propose an intuitive generalization to the Generative Adversarial Networks (GANs) and its conditional variants to address the well known mode collapse problem. Firstly, we propose a multi-agent GAN architecture incorporating multiple generators and one discriminator. Secondly, to enforce different generators to capture diverse high probability modes, we modify discriminator's objective function where along with finding the real and fake samples, the discriminator has to identify the generator that generated the fake sample. Intuitively, to succeed in this task, the discriminator must learn to push different generators towards different identifiable modes. Our framework (MAD-GAN) is generalizable in the sense that it can be easily combined with other existing variants of GANs to produce diverse samples. We perform extensive experiments on synthetic and real datasets and compare MAD-GAN with different variants of GAN. We show high quality diverse sample generations for the challenging tasks such as image-to-image translation (known to learn delta distribution) and face generation. In addition, we show that MAD-GAN is able to disentangle different modalities even when trained using highly challenging multi-view dataset (mixture of forests, icebergs, bedrooms etc). In the end, we also show its efficacy for the unsupervised feature representation task. In the appendix we introduce a similarity based competing objective which encourages the different generators to generate varied samples judged by a user defined similarity metric. We show extensive evaluations on a 1-D setting of mixture of gaussians for non parametric density estimation. The theoretical proofs back the efficacy of the framework and explains why various generators are pushed towards distinct clusters of modes.

The excess risk of a M-estimator is a fundamental quantity of the theory of statistical learning. Consequently, a general theory of rates of convergence as been developed in the nineties and early 2000 ([Mas07, Kol11]). However, it has been recently identified that some theoretical descriptions of learning procedures need finer controls than those brought by the classical upper bounds of the excess risk. In this case, the derivation of concentration inequalities for the excess risk is a new and exiting axis of research, of particular importance for obtaining satisfying oracle inequalities in various contexts, especially linked to high dimension. In the field of model selection, it has been indeed remarked that such concentration inequalities allow to discuss the non-asymptotic optimality of model selection procedures ([BM07, AM09, Ler11, Sau12]).

This paper presents a new approach for Gaussian process (GP) regression for large datasets. The approach involves partitioning the regression input domain into multiple local regions with a different local GP model fitted in each region. Unlike existing local partitioned GP approaches, we introduce a technique for patching together the local GP models nearly seamlessly to ensure that the local GP models for two neighboring regions produce nearly the same response prediction and prediction error variance on the boundary between the two regions. This effectively solves the well-known discontinuity problem that degrades the boundary accuracy of existing local partitioned GP methods. Our main innovation is to represent the continuity conditions as additional pseudo-observations that the differences between neighboring GP responses are identically zero at an appropriately chosen set of boundary input locations. To predict the response at any input location, we simply augment the actual response observations with the pseudo-observations and apply standard GP prediction methods to the augmented data. In contrast to heuristic continuity adjustments, this has an advantage of working within a formal GP framework, so that the GP-based predictive uncertainty quantification remains valid. Our approach also inherits a sparse block-like structure for the sample covariance matrix, which results in computationally efficient closed-form expressions for the predictive mean and variance. In addition, we provide a new spatial partitioning scheme based on a recursive space partitioning along local principal component directions, which makes the proposed approach applicable for regression domains having more than two dimensions. Using three spatial datasets and three higher dimensional datasets, we investigate the numerical performance of the approach and compare it to several state-of-the-art approaches.

In bankruptcy prediction, the proportion of events is very low, which is often oversampled to eliminate this bias. In this paper, we study the influence of the event rate on discrimination abilities of bankruptcy prediction models. First the statistical association and significance of public records and firmographics indicators with the bankruptcy were explored. Then the event rate was oversampled from 0.12% to 10%, 20%, 30%, 40%, and 50%, respectively. Seven models were developed, including Logistic Regression, Decision Tree, Random Forest, Gradient Boosting, Support Vector Machine, Bayesian Network, and Neural Network. Under different event rates, models were comprehensively evaluated and compared based on Kolmogorov-Smirnov Statistic, accuracy, F1 score, Type I error, Type II error, and ROC curve on the hold-out dataset with their best probability cut-offs. Results show that Bayesian Network is the most insensitive to the event rate, while Support Vector Machine is the most sensitive.

As you probably know, machine learning studies computer algorithms to learn how to do stuff. We might, for instance, be interested in learning to complete a task or to make accurate predictions or to behave intelligently. The learning that is being done is always based on some sort of observations or data, such as examples, direct experience, or instruction. So, in general, machine learning is about learning to do better in the future based on what was experienced in the past. Machine learning is being used in a lot of real-world applications for various purposes.

Have you ever thought of working in Artificial Intelligence (AI)? With the advancement in digital technologies, AI and Machine Learning are the buzzwords today. And if you want to build a career in this fast-growing domain, now's the time to start. Join domain expert and Big Data influencer, Ronald Van Loon, and Anand Narayanan, Chief Product Officer at Simplilearn, as they discuss why AI and machine learning are so popular and what skills can help you carve out a career in this field. In this fireside chat, you'll learn about: 1. AI and its various disciplines 2. Some real-life applications of AI and machine learning 3. Career choices in AI and what skills can help you land a job in this domain Don't miss this live 60 minute-long chat about one of the most sought-after domains in the business world today.