Bilinear pooling for fine-grained visual recognition and multi-modal deep learning

Bilinear pooling originated in the computer vision community as a method for fine-grained visual recognition. Or in a less fancy language, a method that looks for specific details when recognizing and classifying visual objects. At a high level, the approach works as follows. Given an input image I, we feed I into two different deep convolutional neural networks A and B, see Figure 1. After applying several pooling and non-linear transformations we output a feature map from both A and B. These two networks might be pretrained in order to solve different tasks.