This paper proposes an out-of-sample extension framework for a global manifold learning algorithm (Isomap) that uses temporal information in out-of-sample points in order to make the embedding more robust to noise and artifacts. Given a set of noise-free training data and its embedding, the proposed framework extends the embedding for a noisy time series. This is achieved by adding a spatio-temporal compactness term to the optimization objective of the embedding. To the best of our knowledge, this is the first method for out-of-sample extension of manifold embeddings that leverages timing information available for the extension set. Experimental results demonstrate that our out-of-sample extension algorithm renders a more robust and accurate embedding of sequentially ordered image data in the presence of various noise and artifacts when compared to other timing-aware embeddings. Additionally, we show that an out-of-sample extension framework based on the proposed algorithm outperforms the state of the art in eye-gaze estimation.

Given an arbitrary dataset, you typically don't know which kernel may work best. I recommend starting with the simplest hypothesis space first -- given that you don't know much about your data -- and work your way up towards the more complex hypothesis spaces. So, the linear kernel works fine if your dataset if linearly separable; however, if your dataset isn't linearly separable, a linear kernel isn't going to cut it (almost in a literal sense;)). For simplicity (and visualization purposes), let's assume our dataset consists of 2 dimensions only. Now, it looks like both linear and RBF kernel SVM would work equally well on this dataset.

With the fast mechanization brought about by the technological revolution, the word manual is slowly getting lost amidst the crowd and will very soon completely vanish. As Big data has whisked the tech industry, Machine Learning is gaining importance and has robustly handled huge amount of data making accurate predictions. In an era of constant progress, we can only guess what astounding invention and discovery is to come next. The data-crunching machines that have been seamlessly executing the advanced techniques. Machine Learning is a subset of the Artificial Intelligence, which is a broader term and concept. Where Artificial Intelligence aims to make computers smarter and intelligent, Machine Learning has come up with ways to do that.

We introduce \texttt{pycobra}, a Python library devoted to ensemble learning (regression and classification) and visualisation. Its main assets are the implementation of several ensemble learning algorithms, a flexible and generic interface to compare and blend any existing machine learning algorithm available in Python libraries (as long as a \texttt{predict} method is given), and visualisation tools such as Voronoi tessellations. \texttt{pycobra} is fully \texttt{scikit-learn} compatible and is released under the MIT open-source license. \texttt{pycobra} can be downloaded from the Python Package Index (PyPi) and Machine Learning Open Source Software (MLOSS). The current version (along with Jupyter notebooks, extensive documentation, and continuous integration tests) is available at \href{https://github.com/bhargavvader/pycobra}{https://github.com/bhargavvader/pycobra}.

Explaining underlying causes or effects about events is a challenging but valuable task. We define a novel problem of generating explanations of a time series event by (1) searching cause and effect relationships of the time series with textual data and (2) constructing a connecting chain between them to generate an explanation. To detect causal features from text, we propose a novel method based on the Granger causality of time series between features extracted from text such as N-grams, topics, sentiments, and their composition. The generation of the sequence of causal entities requires a commonsense causative knowledge base with efficient reasoning. To ensure good interpretability and appropriate lexical usage we combine symbolic and neural representations, using a neural reasoning algorithm trained on commonsense causal tuples to predict the next cause step. Our quantitative and human analysis show empirical evidence that our method successfully extracts meaningful causality relationships between time series with textual features and generates appropriate explanation between them.

Orthogonal matching pursuit (OMP) and orthogonal least squares (OLS) are widely used for sparse signal reconstruction in under-determined linear regression problems. The performance of these compressed sensing (CS) algorithms depends crucially on the \textit{a priori} knowledge of either the sparsity of the signal ($k_0$) or noise variance ($\sigma^2$). Both $k_0$ and $\sigma^2$ are unknown in general and extremely difficult to estimate in under determined models. This limits the application of OMP and OLS in many practical situations. In this article, we develop two computationally efficient frameworks namely TF-IGP and RRT-IGP for using OMP and OLS even when $k_0$ and $\sigma^2$ are unavailable. Both TF-IGP and RRT-IGP are analytically shown to accomplish successful sparse recovery under the same set of restricted isometry conditions on the design matrix required for OMP/OLS with \textit{a priori} knowledge of $k_0$ and $\sigma^2$. Numerical simulations also indicate a highly competitive performance of TF-IGP and RRT-IGP in comparison to OMP/OLS with \textit{a priori} knowledge of $k_0$ and $\sigma^2$.

Sparsity learning with known grouping structures has received considerable attention due to wide modern applications in high-dimensional data analysis. Although advantages of using group information have been well-studied by shrinkage-based approaches, benefits of group sparsity have not been well-documented for greedy-type methods, which much limits our understanding and use of this important class of methods. In this paper, generalizing from a popular forward-backward greedy approach, we propose a new interactive greedy algorithm for group sparsity learning and prove that the proposed greedy-type algorithm attains the desired benefits of group sparsity under high dimensional settings. An estimation error bound refining other existing methods and a guarantee for group support recovery are also established simultaneously. In addition, an interactive feature is incorporated to allow extra algorithm flexibility without compromise in theoretical properties. The promising use of our proposal is demonstrated through numerical evaluations including a real industrial application in human activity recognition.

In this paper we establish fast convergence rate of stochastic variance reduction gradient (SVRG) for a class of problems motivated by applications in high dimensional statistics where the problems are not strongly convex, or even non-convex. High-dimensional statistics has achieved remarkable success in the last decade, including results on consistency and rates for various estimator under non-asymptotic high-dimensional scaling, especially when the problem dimensionp is larger than the number of datan [e.g., Negahban et al., 2009, Cand es and Recht, 2009, and many others [Candes et al., 2006, Wainwright, 2006, Chen et al., 2011]] . It is now well known that while this setup appears ill-posed, the estimation or recovery is indeed possible by exploiting the underlying structure of the parameter space - notable examples include sparse vectors, low-rank matrices, and structured regression functions, among others. Recently, estimators leading to non-convex optimizations have gained fast growing attention. Not only it typically has better statistical properties in the high dimensional regime, but also in contrast to common belief, under many cases there exist efficient algorithms that provably find near-optimal solutions Loh and Wainwright [2011], Zhang and Zhang [2012], Loh and Wainwright [2013] . Computation challenges of statistical estimators and machine learning algorithms have been an active area of study, thanks to countless applications involving big data - datasets where both p and n are large.

The statistical dependencies which independent component analysis (ICA) cannot remove often provide rich information beyond the linear independent components. It would thus be very useful to estimate the dependency structure from data. While such models have been proposed, they usually concentrated on higher-order correlations such as energy (square) correlations. Yet, linear correlations are a most fundamental and informative form of dependency in many real data sets. Linear correlations are usually completely removed by ICA and related methods, so they can only be analyzed by developing new methods which explicitly allow for linearly correlated components. In this paper, we propose a probabilistic model of linear non-Gaussian components which are allowed to have both linear and energy correlations. The precision matrix of the linear components is assumed to be randomly generated by a higher-order process and explicitly parametrized by a parameter matrix. The estimation of the parameter matrix is shown to be particularly simple because using score matching, the objective function is a quadratic form. Using simulations with artificial data, we demonstrate that the proposed method improves identifiability of non-Gaussian components by simultaneously learning their correlation structure. Applications on simulated complex cells with natural image input, as well as spectrograms of natural audio data show that the method finds new kinds of dependencies between the components.

Editor's note: This tutorial series was started in September of 2014, with the 8 installments coming over the course of 2 years. I only mention this to put John's first paragraph into context, and to assure readers that this informative series of tutorials, including all of its code, is as relevant and up-to-date today as it was at the time it was written. This is great material, both for anyone taking Andrew Ng's MOOC and as a standalone resource. One of the pivotal moments in my professional development this year came when I discovered Coursera. I'd heard of the "MOOC" phenomenon but had not had the time to dive in and take a class.