Understanding Deep Self-attention Mechanism in Convolution Neural Networks

In order to implement global reference for each pixel-level prediction, Wang et al. proposed self-attention mechanism in CNN (Figure 1). Their approach is based on covariance between the predicted pixel and every other pixel, in which each pixel is considered as a random variable. The attended target pixel is just a weighted summation of all the pixel values, where the weights are the relationship between each pixel and the target pixel. If we reduce the original Figure 1 to the simplest form as Figure 1, we can easily understand the role covariance plays in the mechanism. Firstly, we have input feature map X with height H and width W. Then we reshape X into three 1-dimensional vectors A, B and C, multiplying A and B to get the covariance matrix with size HWxHW.