Sprinkled throughout Andrew Ng's machine learning class is a lot of practical advice for applying machine learning. That's what I'm trying to compile and summarize here. The key is dividing data into training, cross-validation and test sets. The test set is used only to evaluate performance, not to train parameters or select a model representation. The rationale for this is that training set error is not a good predictor of how well your hypothesis will generalize to new examples.

If I swap positive class and negative class, then train a model again (I tried decision tree, adaboost, svm from scikit-learn built-in package) for a two class classification problem. Sometimes, I can see AUC slightly change (around 1-2%). Anyone have any ideas why there are such changes? For ROC curve, x-axis is false positive rate, and y-axis and true positive rate. When prediction model gives prediction scores, we will order the scores from higher value to lower value, and then choose threshold according to the sorted values and calculate at the specific threshold point, what is the fpr and tpr.

The home feed, a collection of Pins from the people, boards and interests followed, as well as recommendations including Picked for You, is the most heavily user-engaged part of the service, and contributes a large fraction of total repins. The more people Pin, the better Pinterest can get for each person, which puts us in a unique position to serve up inspiration as a discovery engine on an ongoing basis. The home feed is a key way to discover new content, which is valuable to the Pinner, but poses a challenging question. Given the ever increasing number of Pins from various sources, how can we surface the most personalized and relevant Pins? Pinnability is the collective name of the machine learning models we developed to help Pinners find the best content in their home feed.

Large volume of networked streaming event data are becoming increasingly available in a wide variety of applications, such as social network analysis, Internet traffic monitoring and healthcare analytics. Streaming event data are discrete observation occurred in continuous time, and the precise time interval between two events carries a great deal of information about the dynamics of the underlying systems. How to promptly detect changes in these dynamic systems using these streaming event data? In this paper, we propose a novel change-point detection framework for multi-dimensional event data over networks. We cast the problem into sequential hypothesis test, and derive the likelihood ratios for point processes, which are computed efficiently via an EM-like algorithm that is parameter-free and can be computed in a distributed fashion. We derive a highly accurate theoretical characterization of the false-alarm-rate, and show that it can achieve weak signal detection by aggregating local statistics over time and networks. Finally, we demonstrate the good performance of our algorithm on numerical examples and real-world datasets from twitter and Memetracker.

The techniques Microsoft Research used to achieve a new world-best error rate will eventually enhance the Cortana Windows 10 personal assistant. Microsoft claims to have achieved the world's lowest error rate for speech recognition, as the company jostles with Amazon, Apple, Google, and IBM to develop products that understand speech as well as humans can. According to Microsoft, its speech scientists at Microsoft Research have achieved a word error rate (WER) of just 6.3 percent under an industry-standard evaluation, using techniques that will eventually enhance Cortana. The previous lowest error rate was 6.9 percent, achieved by IBM's Watson team, which beat their own record of eight percent set last year. Both Microsoft and IBM presented papers detailing their work on speech recognition at the Interspeech conference in San Francisco this week, where papers were also presented by Google's speech researchers.

We consider the estimation and inference of sparse graphical models that characterize the dependency structure of high-dimensional tensor-valued data. To facilitate the estimation of the precision matrix corresponding to each way of the tensor, we assume the data follow a tensor normal distribution whose covariance has a Kronecker product structure. A critical challenge in the estimation and inference of this model is the fact that its penalized maximum likelihood estimation involves minimizing a non-convex objective function. To address it, this paper makes two contributions: (i) In spite of the non-convexity of this estimation problem, we prove that an alternating minimization algorithm, which iteratively estimates each sparse precision matrix while fixing the others, attains an estimator with the optimal statistical rate of convergence. Notably, such an estimator achieves estimation consistency with only one tensor sample, which was not observed in the previous work. (ii) We propose a de-biased statistical inference procedure for testing hypotheses on the true support of the sparse precision matrices, and employ it for testing a growing number of hypothesis with false discovery rate (FDR) control. The asymptotic normality of our test statistic and the consistency of FDR control procedure are established. Our theoretical results are further backed up by thorough numerical studies. We implement the methods into a publicly available R package Tlasso.

The techniques Microsoft Research used to achieve a new world-best error rate will eventually enhance the Cortana Windows 10 personal assistant. Microsoft claims to have achieved the world's lowest error rate for speech recognition, as the company jostles with Amazon, Apple, Google, and IBM to develop products that understand speech as well as humans can. According to Microsoft, its speech scientists at Microsoft Research have achieved a word error rate (WER) of just 6.3 percent under an industry-standard evaluation, using techniques that will eventually enhance Cortana. The previous lowest error rate was 6.9 percent, achieved by IBM's Watson team, which beat their own record of eight percent set last year. Both Microsoft and IBM presented papers detailing their work on speech recognition at the Interspeech conference in San Francisco this week, where papers were also presented by Google's speech researchers.

In regression settings where explanatory variables have very low correlations and there are relatively few effects, each of large magnitude, we expect the Lasso to find the important variables with few errors, if any. This paper shows that in a regime of linear sparsity---meaning that the fraction of variables with a non-vanishing effect tends to a constant, however small---this cannot really be the case, even when the design variables are stochastically independent. We demonstrate that true features and null features are always interspersed on the Lasso path, and that this phenomenon occurs no matter how strong the effect sizes are. We derive a sharp asymptotic trade-off between false and true positive rates or, equivalently, between measures of type I and type II errors along the Lasso path. This trade-off states that if we ever want to achieve a type II error (false negative rate) under a critical value, then anywhere on the Lasso path the type I error (false positive rate) will need to exceed a given threshold so that we can never have both errors at a low level at the same time. Our analysis uses tools from approximate message passing (AMP) theory as well as novel elements to deal with a possibly adaptive selection of the Lasso regularizing parameter.

When it comes to detecting OS X malware, the future may not be rooted in machine learning algorithms, but patterns and heatmap visualization, a researcher posits. In an academic paper published by Virus Bulletin on Monday, Vincent Van Mieghem, a former student at the Delft University of Technology in the Netherlands, describes how a recurring pattern he observed in OS X system calls can be used to indicate the presence of malware. Van Mieghem wrote the paper, "Behavioral Detection and Prevention of Malware on OS X," (.PDF) while interning at Fox-IT but has since moved on to PricewaterhouseCoopers' cybersecurity division. By the numbers, the detection method Van Mieghem concocted is a success; it detected infections from 100 percent of malware samples found on OS X systems at the time. The method apparently leaves little room for error too; it resulted in a scant 0 percent to 20 percent false positive rate, depending on the user, according to the paper.

In this paper, we study the performance of extremum estimators from the perspective of generalization ability (GA): the ability of a model to predict outcomes in new samples from the same population. By adapting the classical concentration inequalities, we derive upper bounds on the empirical out-of-sample prediction errors as a function of the in-sample errors, in-sample data size, heaviness in the tails of the error distribution, and model complexity. We show that the error bounds may be used for tuning key estimation hyper-parameters, such as the number of folds $K$ in cross-validation. We also show how $K$ affects the bias-variance trade-off for cross-validation. We demonstrate that the $\mathcal{L}_2$-norm difference between penalized and the corresponding un-penalized regression estimates is directly explained by the GA of the estimates and the GA of empirical moment conditions. Lastly, we prove that all penalized regression estimates are $L_2$-consistent for both the $n \geqslant p$ and the $n < p$ cases. Simulations are used to demonstrate key results. Keywords: generalization ability, upper bound of generalization error, penalized regression, cross-validation, bias-variance trade-off, $\mathcal{L}_2$ difference between penalized and unpenalized regression, lasso, high-dimensional data.