Forest Representation Learning Guided by Margin Distribution

In recent years, deep neural networks have achieved excellent performance in many application scenarios such as face recognition and automatic speech recognition (ASR) [21]. However, deep neural networks are difficult to be interpreted. This defect severely restricts the development of deep learning in a few application scenarios, where the model's interpretability is needed. Moreover, the deep neural networks are very data-hungry due to the large complexity of the models, which means that the model's performance can decrease significantly when the size of the training data decreases [12, 22]. In many real tasks, due to the high cost of data collection and labeling, the amount of labeled training data may be insufficient to train a deep neural network. In such a situation, traditional learning methods such as random forest (R.F.) [3], gradient boosting decision tree (GBDT) [15, 5], support-vector machines (SVMs) [7], etc., are still good choices. By realizing that the essence of deep learning lies in the layer-by-layer processing, in-model feature transformation, and sufficient model complexity [33], recently Zhou and Feng [32] proposed the deep forest model and the gcForest algorithm to achieve forest representation learning. It can achieve excellent performance on a broad range of tasks, and can even perform well on small or middle-scale of data. Later on, a more efficient improvement was presented [24], and it shows that forest is able to do auto-encoder which thought to be a specialty of neural networks [13].