ECOC as a Method of Constructing Deep Convolutional Neural Network Ensembles

This capability is attributed to the complementarity of the individual classifiers in an ensemble, which jointly offer an error correcting mechanism, and is manifested in low prediction bias and variance [1, 2, 3, 4]. The combination rules employed by these systems can be as simple as taking a vote between the base classifiers, or more complex, where the classifiers are trained to compensate for the weaknesses of each other. Although several combination techniques such as averaging, majority voting [5], bagging [6], stacking [7], random forests [8], error correcting output coding [9, 10] and their variants have been widely used in traditional machine learning, extensions of most of these approaches to the deep learning (DL) systems have been deemed inefficient and challenging, due to the computational complexity associated with the training of deep networks, as well as the difficulty in securing diversity among the base classifiers. Therefore, most of the state-of-art DL ensembles are either formed of simple averaging (or voting) frameworks comprising only a small number of elements [11, 12, 13, 14, 15], or weak decision tree ensembles based on boosting of already extracted deep features [16, 17, 18, 19]. In the literature, deep base classifiers for averaging ensembles are mainly obtained by varying various DL elements such as the types of network architectures, and their parameters, data augmentation techniques, and learning meta parameters. An example is DeepFace [11], where Taigman et al. construct an ensemble face verification system of 7 deep networks and achieve 97.35% accuracy, compared to 97.0% obtained using a single face verification network.