For the supervised least squares classifier, when the number of training objects is smaller than the dimensionality of the data, adding more data to the training set may first increase the error rate before decreasing it. This, possibly counterintuitive, phenomenon is known as peaking. In this work, we observe that a similar but more pronounced version of this phenomenon also occurs in the semi-supervised setting, where instead of labeled objects, unlabeled objects are added to the training set. We explain why the learning curve has a more steep incline and a more gradual decline in this setting through simulation studies and by applying an approximation of the learning curve based on the work by Raudys & Duin.

Motivation: Recent advances in technology for brain imaging and high-throughput genotyping have motivated studies examining the influence of genetic variation on brain structure. Wang et al. (Bioinformatics, 2012) have developed an approach for the analysis of imaging genomic studies using penalized multi-task regression with regularization based on a novel group $l_{2,1}$-norm penalty which encourages structured sparsity at both the gene level and SNP level. While incorporating a number of useful features, the proposed method only furnishes a point estimate of the regression coefficients; techniques for conducting statistical inference are not provided. A new Bayesian method is proposed here to overcome this limitation. Results: We develop a Bayesian hierarchical modeling formulation where the posterior mode corresponds to the estimator proposed by Wang et al. (Bioinformatics, 2012), and an approach that allows for full posterior inference including the construction of interval estimates for the regression parameters. We show that the proposed hierarchical model can be expressed as a three-level Gaussian scale mixture and this representation facilitates the use of a Gibbs sampling algorithm for posterior simulation. Simulation studies demonstrate that the interval estimates obtained using our approach achieve adequate coverage probabilities that outperform those obtained from the nonparametric bootstrap. Our proposed methodology is applied to the analysis of neuroimaging and genetic data collected as part of the Alzheimer's Disease Neuroimaging Initiative (ADNI), and this analysis of the ADNI cohort demonstrates clearly the value added of incorporating interval estimation beyond only point estimation when relating SNPs to brain imaging endophenotypes.

In an illiquid stock, traders can collude and place orders on a predetermined price and quantity at a fixed schedule. This is usually done to manipulate the price of the stock or to create artificial liquidity in the stock, which may mislead genuine investors. Here, the problem is to identify such group of colluding traders. We modeled the problem instance as a graph, where each trader corresponds to a vertex of the graph and trade corresponds to edges of the graph. Further, we assign weights on edges depending on total volume, total number of trades, maximum change in the price and commonality between two vertices. Spectral clustering algorithms are used on the constructed graph to identify colluding group(s). We have compared our results with simulated data to show the effectiveness of spectral clustering to detecting colluding groups. Moreover, we also have used parameters of real data to test the effectiveness of our algorithm.

In engineering, there are various ways to build a key-value storage, and each design makes a different set of assumptions about the usage pattern. In statistical modeling, there are various algorithms to build a classifier, and each algorithm makes a different set of assumptions about the data. When dealing with small amounts of data, it's reasonable to try as many algorithms as possible and to pick the best one since the cost of experimentation is low. But as we hit "big data", it pays off to analyze the data upfront and then design the modeling pipeline (pre-processing, modeling, optimization algorithm, evaluation, productionization) accordingly. As pointed out in my previous post, there are dozens of ways to solve a given modeling problem. Each model assumes something different, and it's not obvious how to navigate and identify which assumptions are reasonable.

Arun There may be some confusion over terminology here. First, for the sake of clarity, I should mention that I will use the term, "predictor" to indicate those (continuous or categorical) variables that are being used to make predictions via a model, not the one that contains the original known classification. I would usually refer to this as the "target", "response" or "output" variable. Second, I would normally think of discriminant and classification methods as doing the same thing - generating a model that will allow you to predict group/class membership from a set of training data with a known grouping variable. In machine learning this is usually referred to as supervised learning.

Linear regression is the most basic and the most widely used technique in machine learning; yet for all its simplicity, studying it can unlock some of the most important concepts in statistics. If you have a basic undestanding of linear regression expressed as \hat{Y} \theta_0 \theta_1X, but don't have a background in statistics and find statements like "ridge regression is equivalent to the maximum a posteriori (MAP) estimate with a zero-mean Gaussian prior" bewildering, then this post is for you. With a superficial goal of understanding that somewhat obtuse statement, its main objective is to explore the topic, starting from the standard formulation of linear regression, moving on to the probabilistic approach (maximum likelihood formulation) and from there to Bayesian linear regression. I'll use the \theta character throughout to refer to the coefficients (weights) of a regression model, either explicitly broken out as \theta_0 and \theta_1 for intercept and slope respectively, or just \theta referring to the vector of coefficients. I'll usually use the expression \theta Tx_i for the prediction a model gives at x_i, the assumption being that a 1 has been added to the vector of values at x_i . 1 In the single predictor case, we know that the least squares fit is the line that minimizes the sum of the squared distances between observed data and predicted values, i.e. it minimizes the Residual Sum of Squares (RSS): These residuals are pretty important in how we reason about our model.

H2O Sparkling Water supports a wide array of algorithms, and as illustrated above it's easy to chain these functions together with dplyr pipelines. To learn more see the H2O Sparkling Water section.

Summary: With concepts of single-feature linear-regression, cost function, gradient descent (from Part 1), epoch, learn-rate, gradient descent variation (from Part 2) under our belt, we are ready to progress to multi-feature linear regression with TensorFlow (TF). If you are already familiar with matrices and multi-feature linear regression, skip to the end for the multi-feature Tensorflow code cheatsheet, or even skip this entire article. The premise of the previous articles was: given any house size (square meters/sqm), which is the feature, we want to predict the house price (), the outcome. In reality, any prediction relies on multiple features, so we advance from single-feature to 2-feature linear regression; we chose 2 features to keep visualization and comprehension simple, but the concept generalizes to any number of features. We introduce a new feature, 'Rooms' (number of units in the house).

We review the Metropolis algorithm -- a simple Markov Chain Monte Carlo (MCMC) sampling method -- and its application to estimating posteriors in Bayesian statistics. A simple python example is provided. Follow @efavdb Follow us on twitter for new submission alerts! One of the central aims of statistics is to identify good methods for fitting models to data. Notice that if we could solve for this function, we would be able to identify which parameter values are most likely -- those that are good candidates for a fit.

The full code is available as an IPython notebook on github. Assuming a multivariate normal distribution for the residuals in linear regression allows us to construct test statistics and therefore specify uncertainty in our fits. A t-test judges the explanatory power of a predictor in isolation, although the standard error that appears in the calculation of the t-statistic is a function of the other predictors in the model. On the other hand, an F-test is a global test that judges the explanatory power of all the predictors together, and we've seen that parsimony in choosing predictors can improve the quality of the overall regression. We've also seen that multicollinearity can throw off the results of individual t-tests as well as obscure the interpretation of the signs of the fitted coefficients. A symptom of multicollinearity is when none of the individual coefficients are significant but the overall F-test is significant.