From von Neumann to Memory-Augmented Neural Networks

The traditional von Neumann architecture differentiates between a CPU (Central Processing Unit) and three levels of memory: registers -- very fast, but with storage capability limited to a few values; main memory (e.g. RAM)-- faster, with enough storage to accommodate for instructions and data to run a program, and external memory (e.g. Memory-Augmented Neural Networks (MANNs) are differentiable versions of the von Neumann architecture (more on this in the next section). The bulk of the neural network can be thought of as the CPU. Certain architectures like RNNs (Recurrent Neural Networks) have built-in memory that is analogous to the registers, storing short-term information.