Reviews: Nonlocal Neural Networks, Nonlocal Diffusion and Nonlocal Modeling

This paper followed the work of nonlocal neural network to discuss the properties of the diffusion and damping effect by analyzing the spectrum. The trained nonlocal network for image classification on CIFSR-10 data by incorporating nonlocal blocks into the 20-layer PreResNet presented most eigenvalues to be negative and convergence challenges when more blocks were added under certain learning rate and epochs. A rough look at the nonlocal operator representation under steady-state shed light that the output signals of the original nonlocal blocks tend to be damped out (diffused) along iterations by design. A new nonlocal network with namely nonlocal stage component was proposed to help overcome the aforementioned damped out problem by essentially replacing the residual part from weighted sum of the neighboring features to the difference between the neighboring signals and computed signals. Another proposed change is replacing the pairwise affinity function based on updated output to the input feature, which stays the same along the propagation with a stage.