Incremental Boosting Convolutional Neural Network for Facial Action Unit Recognition

Recognizing facial action units (AUs) from spontaneous fac ial expressions is still a challenging problem. Most recently, CNNs have shown promi se on facial AU recognition. However, the learned CNNs are often overfitted and do not generalize well to unseen subjects due to limited AU-coded traini ng images. W e proposed a novel Incremental Boosting CNN (IB-CNN) to integrat e boosting into the CNN via an incremental boosting layer that selects discr iminative neurons from the lower layer and is incrementally updated on success ive mini-batches. In addition, a novel loss function that accounts for errors fro m both the incremental boosted classifier and individual weak classifiers was pr oposed to fine-tune the IB-CNN. Experimental results on four benchmark AU datab ases have demonstrated that the IB-CNN yields significant improvement over the traditional CNN and the boosting CNN without incremental learning, as well a s outperforming the state-of-the-art CNN-based methods in AU recognition. The improvement is more impressive for the AUs that have the lowest frequencies in th e databases.