In our previous post, we discussed what is feature selection and why we need feature selection. In this post, we're going to look at the different methods used in feature selection. There are three main classification of feature selection methods – Filter Methods, Wrapper Methods, and Embedded Methods. We'll look at all of them individually. Filter methods are learning-algorithm-agnostic, which means they can be employed no matter which learning algorithm you're using.

At IBM Research, we are exploring new solutions for a range of health care challenges. One such challenge is emergency room (ER) overcrowding, which can lead to long wait times for treatment. Patients who use the ER for non-emergency situations are more likely to return to the ER multiple times (Poole et al. 2016), further contributing to overcrowding. Identifying those patients who are likely to return to the ER may enable hospitals to intervene to ensure access to necessary care outside the ER and potentially alleviate overcrowding. My team at IBM Research-China took on this challenge.

Most brain disorders are very heterogeneous in terms of their underlying biology and developing analysis methods to model such heterogeneity is a major challenge. A promising approach is to use probabilistic regression methods to estimate normative models of brain function using (f)MRI data then use these to map variation across individuals in clinical populations (e.g., via anomaly detection). To fully capture individual differences, it is crucial to statistically model the patterns of correlation across different brain regions and individuals. However, this is very challenging for neuroimaging data because of high-dimensionality and highly structured patterns of correlation across multiple axes. Here, we propose a general and flexible multi-task learning framework to address this problem. Our model uses a tensor-variate Gaussian process in a Bayesian mixed-effects model and makes use of Kronecker algebra and a low-rank approximation to scale efficiently to multi-way neuroimaging data at the whole brain level. On a publicly available clinical fMRI dataset, we show that our computationally affordable approach substantially improves detection sensitivity over both a mass-univariate normative model and a classifier that --unlike our approach-- has full access to the clinical labels.

Random forests are a powerful method for non-parametric regression, but are limited in their ability to fit smooth signals, and can show poor predictive performance in the presence of strong, smooth effects. Taking the perspective of random forests as an adaptive kernel method, we pair the forest kernel with a local linear regression adjustment to better capture smoothness. The resulting procedure, local linear forests, enables us to improve on asymptotic rates of convergence for random forests with smooth signals, and provides substantial gains in accuracy on both real and simulated data.

Modern measurement technologies provide the means to measure high density spatial and geometric data in three-dimensional (3D) coordinate systems, referred to as point clouds. Point cloud data analysis has broad applications in advanced manufacturing and metrology for measuring dimensional accuracy and shape analysis, in geographic information systems (GIS) for digital elevation modeling and analysis of terrains, in computer graphics for shape reconstruction, and in medical imaging for volumetric measurement to name a few. The role of point cloud data in manufacturing is now more important than ever, particularly in the field of smart and additive manufacturing processes, where products with complex shape and geometry are manufactured with the help of advanced technologies (Gibson et al., 2010). In these processes, the dimensional and geometric accuracy of manufactured parts are measured in the form of point clouds using modern sensing devices, including touch-probe coordinate measuring machines (CMM) and optical systems, such as laser scanners. Modeling the relationship of the dimensional accuracy, encapsulated in point clouds, with process parameters and machine settings is vital for variation reduction and process optimization.

Whenever we look at any online course, they take off with linear regression and this is a concept that most of us know, that is, an equation of a line initially and then gradually fitting of the best fit line. The application of this algorithm is used in machine learning as a way to predict results in the future given the feature vectors, x. So, why is the cost function a squared cost function? Why not have an absolute cost function? Well, there are plenty of reasons as to why we consider this, but when we derive this mathematically, we come across the concept of exponential families under general linear models, which generalize the notion of loss functions for any given model, and thus the square function is actually an exponential family curve.

Multi-view data have been routinely collected in various fields of science and engineering. A general problem is to study the predictive association between multivariate responses and multi-view predictor sets, all of which can be of high dimensionality. It is likely that only a few views are relevant to prediction, and the predictors within each relevant view contribute to the prediction collectively rather than sparsely. We cast this new problem under the familiar multivariate regression framework and propose an integrative reduced-rank regression (iRRR), where each view has its own low-rank coefficient matrix. As such, latent features are extracted from each view in a supervised fashion. For model estimation, we develop a convex composite nuclear norm penalization approach, which admits an efficient algorithm via alternating direction method of multipliers. Extensions to non-Gaussian and incomplete data are discussed. Theoretically, we derive non-asymptotic oracle bounds of iRRR under a restricted eigenvalue condition. Our results recover oracle bounds of several special cases of iRRR including Lasso, group Lasso and nuclear norm penalized regression. Therefore, iRRR seamlessly bridges group-sparse and low-rank methods and can achieve substantially faster convergence rate under realistic settings of multi-view learning. Simulation studies and an application in the Longitudinal Studies of Aging further showcase the efficacy of the proposed methods.

There are no universally agreed-upon definitions for the terms "machine learning", "data science", and "statistics". In my mind, classical statistics consists of traditional techniques that were developed from the 1920s through the 1970s. Statistics techniques include things like correlation, linear regression, and the t-test for hypothesis testing. In my mind, machine learning consists of techniques that make predictions based on data and usually require computer analysis. Examples include logistic regression classification, neural network classification, and k-means clustering.

Whenever we look at any online course, they take off with linear regression and this is a concept that most of us study write from our 8th grades, that is, an equation of a line initially and then gradually fitting of the best fit line. The application of this algorithm is used in machine learning as a way to predict results in the future given the feature vectors, x. So, why is the cost function a squared cost function? Why not have an absolute cost function? Well, there are plenty of reasons as to why we consider this, but when we derive this mathematically, we come across the concept of exponential families under general linear models, which generalize the notion of loss functions for any given model, and thus the square function is actually an exponential family curve.

One week into my Research Science role at Lyft, I merged my first pull request into the Fraud team's code repository and deployed our fraud decision service. No, it wasn't to launch a groundbreaking user behavior activity-based convolutional recurrent neural network trained in a semi-supervised, adversarial fashion that challenges a user to prove her identity -- it would be a couple of years before that. Embarrassingly, it was to remove a duplicate line of feature coefficients in a hand-coded logistic regression model rolled out a little less than a year before. This small bug exposed a number of limitations of a system built primarily for a different type of usage -- that of business rules that encapsulate simple, human-readable handcrafted logic. In our old worldview, models were simply extensions of business rules.