We consider semi-supervised regression when the predictor variables are drawn from an unknown manifold. A simple two step approach to this problem is to: (i) estimate the manifold geodesic distance between any pair of points using both the labeled and unlabeled instances; and (ii) apply a k nearest neighbor regressor based on these distance estimates. We prove that given sufficiently many unlabeled points, this simple method of geodesic kNN regression achieves the optimal finite-sample minimax bound on the mean squared error, as if the manifold were known. Furthermore, we show how this approach can be efficiently implemented, requiring only O(k N log N) operations to estimate the regression function at all N labeled and unlabeled points. We illustrate this approach on two datasets with a manifold structure: indoor localization using WiFi fingerprints and facial pose estimation. In both cases, geodesic kNN is more accurate and much faster than the popular Laplacian eigenvector regressor.

In this study we consider learning a reduced dimensionality representation from datasets obtained under multiple views. Such multiple views of datasets can be obtained, for example, when the same underlying process is observed using several different modalities, or measured with different instrumentation. Our goal is to effectively exploit the availability of such multiple views for various purposes, such as non-linear embedding, manifold learning, spectral clustering, anomaly detection and non-linear system identification. Our proposed method exploits the intrinsic relation within each view, as well as the mutual relations between views. We do this by defining a cross-view model, in which an implied Random Walk process between objects is restrained to hop between the different views. Our method is robust to scaling of each dataset, and is insensitive to small structural changes in the data. Within this framework, we define new diffusion distances and analyze the spectra of the implied kernels. We demonstrate the applicability of the proposed approach on both artificial and real data sets.

Starting from a dataset with input/output time series generated by multiple deterministic linear dynamical systems, this paper tackles the problem of automatically clustering these time series. We propose an extension to the so-called Martin cepstral distance, that allows to efficiently cluster these time series, and apply it to simulated electrical circuits data. Traditionally, two ways of handling the problem are used. The first class of methods employs a distance measure on time series (e.g. Euclidean, Dynamic Time Warping) and a clustering technique (e.g. k-means, k-medoids, hierarchical clustering) to find natural groups in the dataset. It is, however, often not clear whether these distance measures effectively take into account the specific temporal correlations in these time series. The second class of methods uses the input/output data to identify a dynamic system using an identification scheme, and then applies a model norm-based distance (e.g. H2, H-infinity) to find out which systems are similar. This, however, can be very time consuming for large amounts of long time series data. We show that the new distance measure presented in this paper performs as good as when every input/output pair is modelled explicitly, but remains computationally much less complex. The complexity of calculating this distance between two time series of length N is O(N logN).

Supervised machine learning models boast remarkable predictive capabilities. But can you trust your model? Will it work in deployment? What else can it tell you about the world? We want models to be not only good, but interpretable. And yet the task of interpretation appears underspecified. Papers provide diverse and sometimes non-overlapping motivations for interpretability, and offer myriad notions of what attributes render models interpretable. Despite this ambiguity, many papers proclaim interpretability axiomatically, absent further explanation. In this paper, we seek to refine the discourse on interpretability. First, we examine the motivations underlying interest in interpretability, finding them to be diverse and occasionally discordant. Then, we address model properties and techniques thought to confer interpretability, identifying transparency to humans and post-hoc explanations as competing notions. Throughout, we discuss the feasibility and desirability of different notions, and question the oft-made assertions that linear models are interpretable and that deep neural networks are not.

In this article, we will go through the intuition of linear regression and a straightforward implementation of the algorithm. This article is adapted from this booklet, in which you can find the mathematics behind the algorithm as well as detailed explanation and implementation details. Linear regression is a simple yet useful learning algorithm that can be seen as a statistical or an optimization problem. For simple regression, there are optimal analytical solutions; however, for high dimensions problems, there are not. Regression fits a function to a data set, so what we are trying to do is to find a representative function and fit it to our data set.

Spectral clustering is amongst the most popular methods for community detection in graphs. A key step in spectral clustering algorithms is the eigen-decomposition of the $n{\times}n$ graph Laplacian matrix to extract its $k$ leading eigenvectors, where $k$ is the desired number of clusters among $n$ objects. This is prohibitively complex to implement for very large datasets. However, it has recently been shown that it is possible to bypass the eigen-decomposition by computing an approximate spectral embedding through graph filtering of random signals. In this paper, we prove that spectral clustering performed via graph filtering can still recover the planted clusters consistently, under mild conditions. We analyse the effects of sparsity, dimensionality and filter approximation error on the consistency of the algorithm.

A multi-way factor analysis model is introduced for tensor-variate data of any order. Each data item is represented as a (sparse) sum of Kruskal decompositions, a Kruskal-factor analysis (KFA). KFA is nonparametric and can infer both the tensor-rank of each dictionary atom and the number of dictionary atoms. The model is adapted for online learning, which allows dictionary learning on large data sets. After KFA is introduced, the model is extended to a deep convolutional tensor-factor analysis, supervised by a Bayesian SVM. The experiments section demonstrates the improvement of KFA over vectorized approaches (e.g., BPFA), tensor decompositions, and convolutional neural networks (CNN) in multi-way denoising, blind inpainting, and image classification. The improvement in PSNR for the inpainting results over other methods exceeds 1dB in several cases and we achieve state of the art results on Caltech101 image classification.

Recent advances in deep learning (dl; LeCun et al., 2015) have revolutionized the application of machine learning in areas such as computer vision (Krizhevsky et al., 2012), speech recognition (Hinton et al., 2012) and natural language processing (Collobert and Weston, 2008). Although certain kernel-based methods have also been successful in such domains (Cho and Saul, 2009; Mairal et al., 2014), it is still unclear whether these methods can indeed catch up with the recent dl breakthroughs. Aside from the benefits obtained from using compositional representations, we believe that the main components contributing to the success of dl techniques are: (i) their scalability to large datasets and efficient computation via gpus; (ii) their large representational power; and (iii) the use of well-targeted objective functions for the problem at hand. In the kernel world, Gaussian process (gp; Rasmussen and Williams, 2006) models are attractive because they are elegant Bayesian nonparametric approaches to learning from data. Nevertheless, besides the limitations intrinsic to local kernel machines (Bengio et al., 2005), it is clear that gp-based methods have not fully explored the desirable criteria highlighted above. Firstly, with regards to (i) scalability, despite recent advances in inducing-variable approaches and variational inference in gp models (Titsias, 2009; Hensman et al., 2013, 2015a; Dezfouli and Bonilla, 2015), the study of truly large datasets in problems other than regression and the investigation of gpu-based acceleration in gp models are still under-explored areas. We note that these issues are also shared by non-probabilistic kernel methods such as support vector machines (svms; Scholkopf and Smola, 2001). Furthermore, concerning (ii) their representational power, kernel methods have been plagued by the overuse of very limited kernels such as the squared exponential kernel, also known as the radial-basisfunction (rbf) kernel.

Power spectral density (PSD) maps providing the distribution of RF power across space and frequency are constructed using power measurements collected by a network of low-cost sensors. By introducing linear compression and quantization to a small number of bits, sensor measurements can be communicated to the fusion center with minimal bandwidth requirements. Strengths of data- and model-driven approaches are combined to develop estimators capable of incorporating multiple forms of spectral and propagation prior information while fitting the rapid variations of shadow fading across space. To this end, novel nonparametric and semiparametric formulations are investigated. It is shown that PSD maps can be obtained using support vector machine-type solvers. In addition to batch approaches, an online algorithm attuned to real-time operation is developed. Numerical tests assess the performance of the novel algorithms.

The data features that we use to train our machine learning models have a huge influence on the performance we can achieve. Irrelevant or partially relevant features can negatively impact model performance. Feature selection is a process where we automatically select those features in our data that contribute most to the prediction variable or output in which we are interested. Having irrelevant features in our data can decrease the accuracy of many models, especially linear algorithms like linear and logistic regression. We can learn more about feature selection with scikit-learn in the article Feature selection.