Adaptive and Iteratively Improving Recurrent Lateral Connections

Adaptive and Iteratively Improving Recurrent Lateral ConnectionsBarak Battash Lior Wolf Tel Aviv University Facebook AI Research & Tel Aviv University Abstract The current leading computer vision models are typically feed forward neural models, in which the output of one computational block is passed to the next one sequentially. This is in sharp contrast to the organization of the primate visual cortex, in which feedback and lateral connections are abundant. In this work, we propose a computational model for the role of lateral connections in a given block, in which the weights of the block vary dynamically as a function of its activations, and the input from the upstream blocks is iteratively reintroduced. We demonstrate how this novel architectural modification can lead to sizable gains in performance, when applied to visual action recognition without pretraining and that it outperforms the literature architectures with recurrent feedback processing on ImageNet. 1 Introduction Rapid exposure experiments in primates teach us that image recognition occurs as early as the first 100 msec of visual perception, a time budget that suffices only for feed-forward inference, due to the relatively slow nature of biological neurons (Perrett and Oram, 1993; Thorpe et al., 1996). However, anatomical studies have shown that feedback connections are prevalent in the cortex (Douglas and Martin, 2004; Felleman and Essen, 1991). As one striking example, the feedforward input from LGN to V1 in cats constitutes only five percent of the total input to V1, the rest being lateral and feedback connections (Binzegger et al., 2004). In fact, lateral connections, which are projections from a layer to itself, are even more prevalent than feedback connections that project from downstream layers upstream. One possible conjecture would be that feedback (including lateral) connections play roles that are replaced by other mechanisms in the current deep learning literature. For example, they could play a role in training the biological neural network, or they can form attention mechanisms, which are captured by attention (Sermanet et al., 2014) and self-attention (Parikh et al., 2016) blocks in modern neural networks. Similarly, one can claim that such connections are required due to the limitations of the biological computational, but may not be necessary in artificial neural networks, which can be extremely deep (Liao and Poggio, 2016).