However, when the predictor variables are highly correlated then multicollinearity can become a problem. This can cause the coefficient estimates of the model to be unreliable and have high variance. One way to avoid this problem is to instead use principal components regression, which finds M linear combinations (known as "principal components") of the original p predictors and then uses least squares to fit a linear regression model using the principal components as predictors. This tutorial provides a step-by-step example of how to perform principal components regression in R. The easiest way to perform principal components regression in R is by using functions from the pls package. For this example, we'll use the built-in R dataset called mtcars which contains data about various types of cars: For this example we'll fit a principal components regression (PCR) model using hp as the response variable and the following variables as the predictor variables: The following code shows how to fit the PCR model to this data.

This paper proposes a distributionally robust approach to logistic regression. We use the Wasserstein distance to construct a ball in the space of probability distributions centered at the uniform distribution on the training samples. If the radius of this ball is chosen judiciously, we can guarantee that it contains the unknown datagenerating distribution with high confidence. We then formulate a distributionally robust logistic regression model that minimizes a worst-case expected logloss function, where the worst case is taken over all distributions in the Wasserstein ball. We prove that this optimization problem admits a tractable reformulation and encapsulates the classical as well as the popular regularized logistic regression problems as special cases. We further propose a distributionally robust approach based on Wasserstein balls to compute upper and lower confidence bounds on the misclassification probability of the resulting classifier. These bounds are given by the optimal values of two highly tractable linear programs.

Even these ideas are not so novel. For example, the local reparametrization trick is something that we use all the time when we do Variational Bayes (VB) (say in a logistic regression model) and transform high-dimensional integrals into one-dimensional integrals under a Gaussian approximate posterior. For example, if you have a likelihood of the form \prod_{i 1} n \sigma(w T x_i) and apply VB with q(w mu,Sigma), then you end up with a sum of expectations of the form \sum_{i 1} n q(w mu,Sigma) \log \sigma(w T x_i) d w and then the local reparametrization trick is applied to transform each separate (initially high-dimensional integral over the vector w) into a 1-D integral over the univariate standard normal. The authors essentially use this separately for each activation unit and apply stochastic approximation instead of integration. Having said that, I must admit that as far as the stochastic variational inference algorithms are concerned and the related research community (born a couple of years ago!) the use of this local reparametrization trick, as far as I know, is novel and people should know about it because it is useful.

We consider Empirical Risk Minimization (ERM) in the context of stochastic optimization with exp-concave and smooth losses--a general optimization framework that captures several important learning problems including linear and logistic regression, learning SVMs with the squared hinge-loss, portfolio selection and more. In this setting, we establish the first evidence that ERM is able to attain fast generalization rates, and show that the expected loss of the ERM solution in d dimensions converges to the optimal expected loss in a rate of d/n. This rate matches existing lower bounds up to constants and improves by a log n factor upon the state-of-the-art, which is only known to be attained by an online-to-batch conversion of computationally expensive online algorithms.

Recently there has been substantial interest in spectral methods for learning dynamical systems. These methods are popular since they often offer a good tradeoff between computational and statistical efficiency. Unfortunately, they can be difficult to use and extend in practice: e.g., they can make it difficult to incorporate prior information such as sparsity or structure.

Trace regression models have received considerable attention in the context of matrix completion, quantum state tomography, and compressed sensing. Estimation of the underlying matrix from regularization-based approaches promoting low-rankedness, notably nuclear norm regularization, have enjoyed great popularity. In this paper, we argue that such regularization may no longer be necessary if the underlying matrix is symmetric positive semidefinite (spd) and the design satisfies certain conditions. In this situation, simple least squares estimation subject to an spd constraint may perform as well as regularization-based approaches with a proper choice of regularization parameter, which entails knowledge of the noise level and/or tuning. By contrast, constrained least squares estimation comes without any tuning parameter and may hence be preferred due to its simplicity.

We thank the reviewers for their comments and interest. R1 Assigned_Reviewer_1). R2 proposes a baseline method to compare with. Our interpretation of the comment is that in the expression Y - Z t beta _2, R2 uses Z to denote the feature-vector and Y a 0-1 label, so this proposal corresponds to standard least-squares regression (with lasso). Generally, logistic (lasso) regression is preferable for binary responses [1]. As we already evaluated our approach against the latter method (Figure 1b), the proposed comparison seems unnecessary given the space constraints.

This model is known as the single-index model in statistics, and, among other things, it represents a significant generalization of one-bit compressed sensing.

Boosting is a technique in machine learning that has been shown to produce models with high predictive accuracy. One of the most common ways to implement boosting in practice is to use XGBoost, short for "extreme gradient boosting." This tutorial provides a step-by-step example of how to use XGBoost to fit a boosted model in R. For this example we'll fit a boosted regression model to the Boston dataset from the MASS package. This dataset contains 13 predictor variables that we'll use to predict one response variable called mdev, which represents the median value of homes in different census tracts around Boston. We can see that the dataset contains 506 observations and 14 total variables.

The authors design and fit a hierarchical Bayesian model for predicting disease trajectories (i.e., a scalar measure of disease severity measured throughout the course of the disease) for individual patients. The overall model is an additive combination of a a number of terms including: (1) a population-level term, (2) a subpopulation term, (3) an individual term, (4) a GP term for structured errors. Each of these terms is a function of time, which is modeled parametrically in terms of the coefficients on pre-defined basis expansions (linear and/or B-splines). The subpopulation term involves a discrete mixture model, and the individual level term is a Bayesian linear regression. Distributions are chosen to be Gaussian, which makes most steps of inference and learning work out nicely.