Deep Incremental Boosting

AdaBoost [9] is considered a successful Ensemble method and is commonly used in combination with traditional Machine Learning algorithms, especially Boosted Decision Trees [3]. One of the main principles behind it is the additional emphasis given to the so-called hard to classify examples from a training set. Deep Neural Networks have also had great success on many visual problems, and there are a number of benchmark datasets in this area where the state-of-the-art results are held by some Deep Learning algorithm [12, 4]. Ideas from Transfer of Learning have found applications in Deep Learning; for example, in Convolutional Neural Networks (CNNs), when sub-features learned early in the training process can be carried forward to a new CNN in order to improve generalisation on a new problem of the same domain [13]. It has also been shown that these Transfer of Learning methods reduce the "warm-up" phase of the training, where a randomly-initialised CNN would have to relearn basic feature selectors from scratch.