A Taxonomy for Neural Memory Networks

Memory has a pivotal role in human cognition and many different types are well known and intensively studied[1]. In neural networks and signal processing the use of memory is concentrated in preserving in some form (by storing past samples or using a state model) the information from the past. A system is said to include memory if the system's output is a function of the current and past samples. Feedforward neural networks are memoryless, but the time delay neural network [2], the gamma neural model [3] and recurrent neural networks are memory networks. An important theoretical result showed that these networks are universal in the space of myopic functions [4]. A methodology to quantify linear memories was presented in [3], which proposed an analytic expression for the compromise between memory depth (how much the past is remembered) and memory resolution (how specifically the system remembers a past event). A similar compromise exists for nonlinear dynamic memories (i.e. using nonlinear state variables to represent the past), but is depends on the type of nonlinearity and there is no known close form solution. It is fair to say that currently the most utilized neural memory is the recurrent neural networks (RNN) for sequence learning. Compared to the time delay neural network, RNN keeps a processed version of the past signal in its state.