Multi-output regression models must exploit dependencies between outputs to maximise predictive performance. The application of Gaussian processes (GPs) to this setting typically yields models that are computationally demanding and have limited representational power. We present the Gaussian Process Autoregressive Regression (GPAR) model, a scalable multi-output GP model that is able to capture nonlinear, possibly input-varying, dependencies between outputs in a simple and tractable way: the product rule is used to decompose the joint distribution over the outputs into a set of conditionals, each of which is modelled by a standard GP. GPAR's efficacy is demonstrated on a variety of synthetic and real-world problems, outperforming existing GP models and achieving state-of-the-art performance on the tasks with existing benchmarks.

We propose a stepsize adaptation scheme for stochastic gradient descent. It operates directly with the loss function and rescales the gradient in order to make fixed predicted progress on the loss. We demonstrate its capabilities by strongly improving the performance of Adam and Momentum optimizers. The enhanced optimizers with default hyperparameters consistently outperform their constant stepsize counterparts, even the best ones, without a measurable increase in computational cost. The performance is validated on multiple architectures including ResNets and the Differential Neural Computer. A prototype implementation as a TensorFlow optimizer is released.

Modern stochastic optimization methods often rely on uniform sampling which is agnostic to the underlying characteristics of the data. This might degrade the convergence by yielding estimates that suffer from a high variance. A possible remedy is to employ non-uniform importance sampling techniques, which take the structure of the dataset into account. In this work, we investigate a recently proposed setting which poses variance reduction as an online optimization problem with bandit feedback. We devise a novel and efficient algorithm for this setting that finds a sequence of importance sampling distributions competitive with the best fixed distribution in hindsight, the first result of this kind. While we present our method for sampling datapoints, it naturally extends to selecting coordinates or even blocks of thereof. Empirical validations underline the benefits of our method in several settings.

Algebraic topology methods have recently played an important role for statistical analysis with complicated geometric structured data. Among them, persistent homology is a well-known tool to extract robust topological features, and outputs as persistence diagrams. Unfortunately, persistence diagrams are point multi-sets which can not be used in machine learning algorithms for vector data. To deal with it, an emerged approach is to use kernel methods. Besides that, geometry for persistence diagrams is also an important factor. A popular geometry for persistence diagrams is the Wasserstein metric. However, Wasserstein distance is not negative definite. Thus, it is limited to build positive definite kernels upon the Wasserstein distance without approximation. In this work, we explore an alternative Riemannian manifold geometry, namely the Fisher information metric. By building upon the geodesic distance on the Riemannian manifold, we propose a positive definite kernel, namely Riemannian manifold kernel. Then, we analyze eigensystem of the integral operator induced by the proposed kernel for kernel machines. Based on that, we conduct generalization error bounds via covering numbers and Rademacher averages for kernel machines using the Riemannian manifold kernel. Additionally, we also show some nice properties for the proposed kernel such as stability, infinite divisibility and comparative time complexity with other kernels for persistence diagrams in term of computation. Throughout experiments with many different tasks on various benchmark datasets, we illustrate that the Riemannian manifold kernel improves performances of other baseline kernels.

These perturbations can be designed to be barely noticeable to the human eye, but can cause large classification errors in state of the art deep networks. While it is tempting to speculate that this vulnerability arises from the complex, nonlinear nature of deep networks, a more plausible explanation is that it is due to the excessive linearity of such networks [3-6]. When we take a linear combination of the components of a high-dimensional input, small, adversarially chosen, perturbations of each component can add up to a large perturbation at the output. Complex operations such as a rectified linear unit (ReLU) operating beyond its bias, or a sigmoid in its linear region, together with operations such as max pooling or average pooling, when cascaded through multiple stages, still amount to an approximately linear combination of the input. Of course, the coefficients of the linear combination exhibit some dependence on the input, but these can be viewed as on-off switches rather than a change in the value of the coefficients: for example, whether the input is such that a ReLU unit is operating in its linear region, or the identity of the argument of the maximum in a max pooling unit. This motivates us to take a step back in this paper, and study adversarial perturbations in the simplest possible setting: a linear classifier.

We consider in this paper the problem of sampling a high-dimensional probability distribution $\pi$ having a density \wrt\ the Lebesgue measure on $\mathbb{R}^d$, known up to a normalization factor $x \mapsto \pi(x)= \mathrm{e}^{-U(x)}/\int_{\mathbb{R}^d} \mathrm{e}^{-U(y)} \mathrm{d}y$. Such problem naturally occurs for example in Bayesian inference and machine learning. Under the assumption that $U$ is continuously differentiable, $\nabla U$ is globally Lipschitz and $U$ is strongly convex, we obtain non-asymptotic bounds for the convergence to stationarity in Wasserstein distance of order $2$ and total variation distance of the sampling method based on the Euler discretization of the Langevin stochastic differential equation, for both constant and decreasing step sizes. The dependence on the dimension of the state space of the obtained bounds is studied to demonstrate the applicability of this method. The convergence of an appropriately weighted empirical measure is also investigated and bounds for the mean square error and exponential deviation inequality are reported for functions which are measurable and bounded. An illustration to Bayesian inference for binary regression is presented.

In ensemble methods, the outputs of a collection of diverse classifiers are combined in the expectation that the global prediction be more accurate than the individual ones. Heterogeneous ensembles consist of predictors of different types, which are likely to have different biases. If these biases are complementary, the combination of their decisions is beneficial. In this work, a family of heterogeneous ensembles is built by pooling classifiers from M homogeneous ensembles of different types of size T. Depending on the fraction of base classifiers of each type, a particular heterogeneous combination in this family is represented by a point in a regular simplex in M dimensions. The M vertices of this simplex represent the different homogeneous ensembles. A displacement away from one of these vertices effects a smooth transformation of the corresponding homogeneous ensemble into a heterogeneous one. The optimal composition of such heterogeneous ensemble can be determined using cross-validation or, if bootstrap samples are used to build the individual classifiers, out-of-bag data. An empirical analysis of such combinations of bootstraped ensembles composed of neural networks, SVMs, and random trees (i.e. from a standard random forest) illustrates the gains that can be achieved by this heterogeneous ensemble creation method.

A particular event in 2008 was billed as a landmark in automotive engineering and received global media coverage. With 31 design and 37 technology patents filed, it was touted as a global land mark in affordable transportation. In terms of estimated market impact it was placed right beside the iconic Ford Model T and VW Beetle. The car was expected to revolutionize personnel transportation in the developing world, however much to disbelief initial sales were dismal and continued to remain a fraction of original expectations. A through analysis years later identified the main culprit.

This is the first article of a multi-part series on using Python and Machine Learning to build models to predict weather temperatures based off data collected from Weather Underground. The series will be comprised of three different articles describing the major aspects of a Machine Learning project. The data used in this series will be collected from Weather Underground's free tier API web service. I will be using the requests library to interact with the API to pull in weather data since 2015 for the city of Lincoln, Nebraska. Once collected, the data will need to be process and aggregated into a format that is suitable for data analysis, and then cleaned. The second article will focus on analyzing the trends in the data with the goal of selecting appropriate features for building a Linear Regression model using the statsmodels and scikit-learn Python libraries. I will discuss the importance of understanding the assumptions necessary for using a Linear Regression model and demonstrate how to evaluate the features to build a robust model.

Learning-to-rank techniques have proven to be extremely useful for prioritization problems, where we rank items in order of their estimated probabilities, and dedicate our limited resources to the top-ranked items. This work exposes a serious problem with the state of learning-to-rank algorithms, which is that they are based on convex proxies that lead to poor approximations. We then discuss the possibility of "exact" reranking algorithms based on mathematical programming. We prove that a relaxed version of the "exact" problem has the same optimal solution, and provide an empirical analysis.