Faster Boosting with Smaller Memory

Neural Information Processing Systems

State-of-the-art implementations of boosting, such as XGBoost and LightGBM, can process large training sets extremely fast. However, this performance requires that the memory size is sufficient to hold a 2-3 multiple of the training set size. This paper presents an alternative approach to implementing the boosted trees, which achieves a significant speedup over XGBoost and LightGBM, especially when the memory size is small. This is achieved using a combination of three techniques: early stopping, effective sample size, and stratified sampling. Our experiments demonstrate a 10-100 speedup over XGBoost when the training data is too large to fit in memory.

XGBoost: Enhancement Over Gradient Boosting Machines


XGBoost was originally developed by Tianqi Chen in his paper titeled "XGBoost: A Scalable Tree Boosting System." XGBoost itself is an enhancement to the gradient boosting algorithm created by Jerome H. Friedman in his paper titled "Greedy Function Approximation: A Gradient Boosting Machine." Both papers are well worth exploring.

Predicting marketing performance with Machine Learning


It's difficult for companies to judge how a marketing campaign has performed in the past never mind predicting how one will perform in the future. This becomes increasingly more difficult when multiple types of campaigns are ran over the same period, different products are available or unexpected outside factors influence user behaviour. In this project I will attempt, using multiple machine learning models, to judge how marketing campaigns have performed and predict how they will perform in the future. For the modelling I will be using a synthetic dataset containing information on Starbucks members over a month long period. Due to small size of this dataset there are a number of limitations with the data that I will outline later in this post.

XGBoost: An Intuitive Explanation


We all know how XGBoost dominates in Kaggle competitions due to its performance and speed. This blog is about understanding how XGBoost works (try to explain the research paper). This blog is not about how to code/ implement XGboost or how to tune its hyperparameters. XGBoost stands for eXtreme Gradient Boosting. It explains bagging (bootstrap aggregating) and boosting (Adaptive Boosting).

Machine Learning Transition Temperatures from 2D Structure


A priori knowledge of melting and boiling could expedite the discovery of pharmaceutical, energetic, and energy harvesting materials. The tools of data science are becoming increasingly important for exploring chemical datasets and predicting material properties. A fundamental part of data-driven modeling is molecular featurization. Herein, we propose a molecular representation with group-constitutive and geometrical descriptors that map to enthalpy and entropy–two thermodynamic quantities that drive phase transitions. The descriptors are inspired by the linear regression-based quantitative structure-property relationship of Yalkowsky and coworkers known as the Unified Physicochemical Property Estimation Relationships (UPPER).

Productionizing Distributed XGBoost to Train Deep Tree Models with Large Data Sets at Uber


Zero values in SparseVectors are treated by XGBoost on Apache Spark as missing values (defaults to Float.NaN) whereas zeroes in DenseVectors are simply treated as zeros. Vector storage in Apache Spark ML is implicitly optimized, so a vector array is stored as a SparseVector or DenseVector based on space efficiency. If an ML practitioner tries to feed a DenseVector at inference time to a model that is trained on SparseVector or vice versa, XGBoost does not provide any warning and the prediction input will likely go into unexpected branches due to the way zeroes are stored, resulting in inconsistent predictions. Hence, it is critical that the storage structure input remains consistent between serving and training times.

Feature Engineering Combined with 1 D Convolutional Neural Network for Improved Mortality Prediction

arXiv.org Machine Learning

The intensive care units (ICUs) are responsible for generating a wealth of useful data in the form of Electronic Health Record (EHR). This data allows for the development of a prediction tool with perfect knowledge backing. We aimed to build a mortality prediction model on 2012 Physionet Challenge mortality prediction database of 4000 patients admitted in ICU. The challenges in the dataset, such as high dimensionality, imbalanced distribution, and missing values were tackled with analytical methods and tools via feature engineering and new variable construction. The objective of the research is to utilize the relations among the clinical variables and construct new variables which would establish the effectiveness of 1-Dimensional Convolutional Neural Network (1- D CNN) with constructed features. Its performance with the traditional machine learning algorithms like XGBoost classifier, Support Vector Machine (SVM), K-Neighbours Classifier (K-NN), and Random Forest Classifier (RF) is compared for Area Under Curve (AUC). The investigation reveals the best AUC of 0.848 using 1-D CNN model.

A Model-driven and Data-driven Fusion Framework for Accurate Air Quality Prediction

arXiv.org Machine Learning

Air quality is closely related to public health. Health issues such as cardiovascular diseases and respiratory diseases, may have connection with long exposure to highly polluted environment. Therefore, accurate air quality forecasts are extremely important to those who are vulnerable. To estimate the variation of several air pollution concentrations, previous researchers used various approaches, such as the Community Multiscale Air Quality model (CMAQ) or neural networks. Although CMAQ model considers a coverage of the historic air pollution data and meteorological variables, extra bias is introduced due to additional adjustment. In this paper, a combination of model-based strategy and data-driven method namely the physical-temporal collection(PTC) model is proposed, aiming to fix the systematic error that traditional models deliver. In the data-driven part, the first components are the temporal pattern and the weather pattern to measure important features that contribute to the prediction performance. The less relevant input variables will be removed to eliminate negative weights in network training. Then, we deploy a long-short-term-memory (LSTM) to fetch the preliminary results, which will be further corrected by a neural network (NN) involving the meteorological index as well as other pollutants concentrations. The data-set we applied for forecasting is from January 1st, 2016 to December 31st, 2016. According to the results, our PTC achieves an excellent performance compared with the baseline model (CMAQ prediction, GRU, DNN and etc.). This joint model-based data-driven method for air quality prediction can be easily deployed on stations without extra adjustment, providing results with high-time-resolution information for vulnerable members to prevent heavy air pollution ahead.

RoNGBa: A Robustly Optimized Natural Gradient Boosting Training Approach with Leaf Number Clipping

arXiv.org Machine Learning

Natural gradient has been recently introduced to the field of boosting to enable the generic probabilistic predication capability. Natural gradient boosting shows promising performance improvements on small datasets due to better training dynamics, but it suffers from slow training speed overhead especially for large datasets. W e present a replication study of NGBoost ( Duan et al., 2019) training that carefully examines the impacts of key hyper-parameters under the circumstance of best-first decision tree learning. W e find that with the regularization of leaf number clipping, the performance of NGBoost can be largely improved via a better choice of hyperparameters. Experiments show that our approach significantly beats the state-of-the-art performance on various kinds of datasets from the UCI Machine Learning Repository while still has up to 4.85x speed up compared with the original approach of NGBoost.

XGBOD: Improving Supervised Outlier Detection with Unsupervised Representation Learning

arXiv.org Machine Learning

A new semi-supervised ensemble algorithm called XGBOD (Extreme Gradient Boosting Outlier Detection) is proposed, described and demonstrated for the enhanced detection of outliers from normal observations in various practical datasets. The proposed framework combines the strengths of both supervised and unsupervised machine learning methods by creating a hybrid approach that exploits each of their individual performance capabilities in outlier detection. XGBOD uses multiple unsupervised outlier mining algorithms to extract useful representations from the underlying data that augment the predictive capabilities of an embedded supervised classifier on an improved feature space. The novel approach is shown to provide superior performance in comparison to competing individual detectors, the full ensemble and two existing representation learning based algorithms across seven outlier datasets.