XGBoost is a library designed and optimized for tree boosting. Gradient boosting trees model is originally proposed by Friedman et al. By embracing multi-threads and introducing regularization, XGBoost delivers higher computational power and more accurate prediction. More than half of the winning solutions in machine learning challenges hosted at Kaggle adopt XGBoost (Incomplete list). XGBoost has provided native interfaces for C, R, python, Julia and Java users.

Last month's post "5 Machine Learning Projects You Can No Longer Overlook" was a well-received piece on 5 lesser-known machine learning projects in the Python ecosystem, and included deep learning libraries, along with auxiliary support, data cleaning, and automation tools. As such, we thought it may be worth doing a follow-up post, but broadening our scope this time. This post will showcase 5 machine learning projects that you may not yet have heard of. This time, however, the projects will include those from across a number of different ecosystems and programming languages, as opposed to focusing solely on Python tools. You may find that, even if you have no requirement for any of these particular tools, inspecting their broad implementation details or their specific code may help in generating some ideas of your own.

With unsupervised learning, the idea of class attributes and explicit class membership does not exist; in fact, one of the dominant forms of unsupervised learning -- data clustering -- aims to approximate class membership by minimizing interclass instance similarity and maximizing intraclass similarity. We will have a look at 2 particular popular methods for attempting to answer this question: the elbow method and the silhouette method. It should be self-evident that, in order to plot this variance against varying numbers of clusters, varying numbers of clusters must be tested. The silhouette method measures the similarity of an object to its own cluster -- called cohesion -- when compared to other clusters -- called separation.

I would probably spend the time on simple (yet useful) algorithms that are representative of these fields (and maybe save reinforcement learning for later).

In this paper, we propose hybrid principal component analysis (HPCA) to extract appearance feature of a face and inter-age group variation-based classifier (IAGVC) with regression to estimate age of a person. The proposed age estimation system is robust and less sensitive to outliers where nonuniform distribution of images at different age groups is existing. Under HPCA, we introduce two novel methods, extended SpPCA and extended SubXPCA. The issues, such as summarization of variance, variable component selection, computational complexity and classification accuracy of HPCA, have been addressed as well. The proposed HPCA operates on subpattern and whole pattern at a time and extracts appearance feature based on both local and global variation of faces.

We consider the problem of estimation of a low-rank matrix from a limited number of noisy rank-one projections. In particular, we propose two fast, non-convex \emph{proper} algorithms for matrix recovery and support them with rigorous theoretical analysis. We show that the proposed algorithms enjoy linear convergence and that their sample complexity is independent of the condition number of the unknown true low-rank matrix. By leveraging recent advances in low-rank matrix approximation techniques, we show that our algorithms achieve computational speed-ups over existing methods. Finally, we complement our theory with some numerical experiments.

In this paper we study stochastic quasi-Newton methods for nonconvex stochastic optimization, where we assume that noisy information about the gradients of the objective function is available via a stochastic first-order oracle (SFO). We propose a general framework for such methods, for which we prove almost sure convergence to stationary points and analyze its worst-case iteration complexity. When a randomly chosen iterate is returned as the output of such an algorithm, we prove that in the worst-case, the SFO-calls complexity is $O(\epsilon^{-2})$ to ensure that the expectation of the squared norm of the gradient is smaller than the given accuracy tolerance $\epsilon$. We also propose a specific algorithm, namely a stochastic damped L-BFGS (SdLBFGS) method, that falls under the proposed framework. {Moreover, we incorporate the SVRG variance reduction technique into the proposed SdLBFGS method, and analyze its SFO-calls complexity. Numerical results on a nonconvex binary classification problem using SVM, and a multiclass classification problem using neural networks are reported.

We consider the stochastic composition optimization problem proposed in \cite{wang2017stochastic}, which has applications ranging from estimation to statistical and machine learning. We propose the first ADMM-based algorithm named com-SVR-ADMM, and show that com-SVR-ADMM converges linearly for strongly convex and Lipschitz smooth objectives, and has a convergence rate of $O( \log S/S)$, which improves upon the $O(S^{-4/9})$ rate in \cite{wang2016accelerating} when the objective is convex and Lipschitz smooth. Moreover, com-SVR-ADMM possesses a rate of $O(1/\sqrt{S})$ when the objective is convex but without Lipschitz smoothness. We also conduct experiments and show that it outperforms existing algorithms.

We propose a unified framework to speed up the existing stochastic matrix factorization (SMF) algorithms via variance reduction. Our framework is general and it subsumes several well-known SMF formulations in the literature. We perform a non-asymptotic convergence analysis of our framework and derive computational and sample complexities for our algorithm to converge to an $\epsilon$-stationary point in expectation. In addition, extensive experiments for a wide class of SMF formulations demonstrate that our framework consistently yields faster convergence and a more accurate output dictionary vis-\`a-vis state-of-the-art frameworks.

Kaggle's 2017 March Machine Learning Mania competition challenged Kagglers to do what millions of sports fans do every year–try to predict the winners and losers of the US men's college basketball tournament. In this winner's interview, 1st place winner, Andrew Landgraf, describes how he cleverly analyzed his competition to optimize his luck. I am interested in sports analytics and have followed the previous competitions on Kaggle. Reading last year's winner's interview, I realized that luck is a major component of winning this competition, just like all brackets. I wanted to see if there was a way of maximizing my luck.