A Beginner's Guide to Attention Mechanisms and Memory Networks

Attention mechanisms in neural networks are about memory access. That's the first thing to remember about attention: it's something of a misnomer. This YouTube video compares attention mechanism to convolutional networks and recurrent neural networks, explains which problems attention solves better than those other algorithms, and then shows how various attention mechanisms are implemented in Deeplearning4j. "Attention" is defined as the "active direction of the mind to an object", or more literarily as "giving heed".1 The word describes the mind's ability to allocate consideration unevenly across a field of sensation, thought and proprioception, to focus and bring certain inputs to the fore, while ignoring or diminishing the importance of others.