Residual Likelihood Forests

Ensemble and Boosting methods such as Random Forests [3] and AdaBoost [19] are often recognized as some of the best out-of-the-box classifiers, consistently achieving state-ofthe-art performance across a wide range of computer vision tasks including applications in image classification [1], semantic segmentation [22], object recognition [12] and data clustering [16]. The success of these methods is attributed to their ability to learn models (strong learners) which possess low bias and variance through the combination of weakly correlated learners (weak learners). Forests reduce variance through averaging its weak learners over the ensemble. Boosting, on the other hand, looks towards reducing both bias and variance through sequentially optimizing under conditional constraints. The commonality between both approaches is in the way each learner is constructed: both methods use a top-down induction algorithm (such as CART [4]) which greedily learns decision nodes in a recursive manner. This approach is known to be suboptimal in terms of objective maximization as there are no guarantees that a global loss is being minimized [14]. In practice, this type of optimization requires the non-linearity offered by several (very) deep trees, which results in redundancy in learned models with large overlaps of information between weak learners. To address these limitations, the ensemble approaches of [11, 20] have utilized gradient information within a boosting framework. This allows weak learners to be fit via pseudoresiduals or to a set of adaptive weights and allows for the minimization of a global loss via gradient descent.