Gaussian process (GP) regression is a flexible, nonparametric approach to regression that naturally quantifies uncertainty. In many applications, the number of responses and covariates are both large, and a goal is to select covariates that are related to the response. For this setting, we propose a novel, scalable algorithm, coined VGPR, which optimizes a penalized GP log-likelihood based on the Vecchia GP approximation, an ordered conditional approximation from spatial statistics that implies a sparse Cholesky factor of the precision matrix. We traverse the regularization path from strong to weak penalization, sequentially adding candidate covariates based on the gradient of the log-likelihood and deselecting irrelevant covariates via a new quadratic constrained coordinate descent algorithm. We propose Vecchia-based mini-batch subsampling, which provides unbiased gradient estimators. The resulting procedure is scalable to millions of responses and thousands of covariates. Theoretical analysis and numerical studies demonstrate the improved scalability and accuracy relative to existing methods.

Recent studies of gradient descent with large step sizes have shown that there is often a regime with an initial increase in the largest eigenvalue of the loss Hessian (progressive sharpening), followed by a stabilization of the eigenvalue near the maximum value which allows convergence (edge of stability). These phenomena are intrinsically non-linear and do not happen for models in the constant Neural Tangent Kernel (NTK) regime, for which the predictive function is approximately linear in the parameters. As such, we consider the next simplest class of predictive models, namely those that are quadratic in the parameters, which we call second-order regression models. For quadratic objectives in two dimensions, we prove that this second-order regression model exhibits progressive sharpening of the NTK eigenvalue towards a value that differs slightly from the edge of stability, which we explicitly compute. In higher dimensions, the model generically shows similar behavior, even without the specific structure of a neural network, suggesting that progressive sharpening and edge-of-stability behavior aren't unique features of neural networks, and could be a more general property of discrete learning algorithms in high-dimensional non-linear models.

Sufficient dimension reduction is a powerful tool to extract core information hidden in the high-dimensional data and has potentially many important applications in machine learning tasks. However, the existing nonlinear sufficient dimension reduction methods often lack the scalability necessary for dealing with large-scale data. We propose a new type of stochastic neural network under a rigorous probabilistic framework and show that it can be used for sufficient dimension reduction for large-scale data. The proposed stochastic neural network is trained using an adaptive stochastic gradient Markov chain Monte Carlo algorithm, whose convergence is rigorously studied in the paper as well. Through extensive experiments on real-world classification and regression problems, we show that the proposed method compares favorably with the existing state-of-the-art sufficient dimension reduction methods and is computationally more efficient for large-scale data.

A coreset is a small set that can approximately preserve the structure of the original input data set. Therefore we can run our algorithm on a coreset so as to reduce the total computational complexity. Conventional coreset techniques assume that the input data set is available to process explicitly. However, this assumption may not hold in real-world scenarios. In this paper, we consider the problem of coresets construction over relational data. Namely, the data is decoupled into several relational tables, and it could be very expensive to directly materialize the data matrix by joining the tables. We propose a novel approach called ``aggregation tree with pseudo-cube'' that can build a coreset from bottom to up. Moreover, our approach can neatly circumvent several troublesome issues of relational learning problems [Khamis et al., PODS 2019]. Under some mild assumptions, we show that our coreset approach can be applied for the machine learning tasks, such as clustering, logistic regression and SVM.

Logistic regression is a method to create a model by using binary data (0 and 1). The goal is to predict something independent variable based on a dependent variable. In a real application, logistic regression is applied to predict the number of customers who buy a product or who did not base on their previous transaction, to predict the number of fraud transactions in credit cards, and so on. In logistic regression, Y-axis lies from 0 – 1. Logistic regression cannot be solved by using a linear equation like linear regression. That is because if the Y-axis of the logistic function is transformed into a linear function, the boundary of the Y-axis lies from -infinity to infinity. Then, when we calculate the misfit error between actual data and predicted data, it will not get a good misfit error.

Robust and sparse estimation of linear regression coefficients is investigated. The situation addressed by the present paper is that covariates and noises are sampled from heavy-tailed distributions, and the covariates and noises are contaminated by malicious outliers. Our estimator can be computed efficiently. Further, the error bound of the estimator is nearly optimal.

Few-shot learning (Fei-Fei et al., 2006) aims to learn a new classification or regression model on a novel task that is not seen during training, given only a few examples in the novel task. Existing few-shot learning methods either rely on episodic meta-learning (Finn et al., 2017, Snell et al., 2017) or standard pretraining (Chen et al., 2019, Tian et al., 2020b) in a supervised manner to extract transferrable knowledge to a new few-shot task. Unfortunately, these methods require many labeled meta-training samples. Acquiring a lot of labeled data is costly or even impossible in practice. Recently, several unsupervised meta-learning approaches have attempted to address this problem by constructing synthetic tasks on unlabeled meta-training data (Hsu et al., 2019, Khodadadeh et al., 2019, 2021) or meta-training on self-supervised pretrained features (Lee et al., 2021a). However, the performance of unsupervised meta-learning approaches is still far from their supervised counterparts. Empirical studies in supervised pretraining show that representation learning via grouping similar samples together (Chen et al., 2019, Dhillon et al., 2020, Laenen and Bertinetto, 2021, Tian et al., 2020b) outperforms a wide range of episodic meta-learning methods, where the definition of similar samples is given by class labels. The motivation of this study is to develop an unsupervised representation learning method by grouping unlabeled meta-training data without episodic training and close the performance gap between supervised and unsupervised few-shot learning. Contrastive self-supervised learning has shown remarkable success in learning representation from unlabeled data, which is competitive with supervised learning on multiple visual tasks (Hénaff et al., 2020, Tian et al., 2020a).

Safe path planning is critical for bipedal robots to operate in safety-critical environments. Common path planning algorithms, such as RRT or RRT*, typically use geometric or kinematic collision check algorithms to ensure collision-free paths toward the target position. However, such approaches may generate non-smooth paths that do not comply with the dynamics constraints of walking robots. It has been shown that the control barrier function (CBF) can be integrated with RRT/RRT* to synthesize dynamically feasible collision-free paths. Yet, existing work has been limited to simple circular or elliptical shape obstacles due to the challenging nature of constructing appropriate barrier functions to represent irregular-shaped obstacles. In this paper, we present a CBF-based RRT* algorithm for bipedal robots to generate a collision-free path through complex space with polynomial-shaped obstacles. In particular, we used logistic regression to construct polynomial barrier functions from a grid map of the environment to represent arbitrarily shaped obstacles. Moreover, we developed a multi-step CBF steering controller to ensure the efficiency of free space exploration. The proposed approach was first validated in simulation for a differential drive model, and then experimentally evaluated with a 3D humanoid robot, Digit, in a lab setting with randomly placed obstacles.

How do we need to detect outliers? How do we need to treat the outliers? An outlier is that datapoint or observation which behaves very differently from the rest of the data. If we are finding the average net worth of a group of people, and if we find Elon Musk in that group, then the complete analysis will go wrong because of just one outlier. This is a reason why outliers should be treated properly before building a machine learning model.

Data science projects fail, frequently. Between the end of 2017 and 2019 several published reports from Gartner, NewVantage, and VentureBeat AI showed that'failure' rates on data science projects are north of 75%. But I don't think this is indicative of how powerful the growth of data, machine learning, and AI has been for business (and likely all sectors of the economy) over the same timeframe. Back-of-the-envelope machine learning is inconspicuously powering business today (2020). A premortem is a thought exercise to predict or foresee why an analysis or project might fail.