Generic construction of scale-invariantly coarse grained memory

Representing the information from the recent past as transient activity distributed over a network has been actively researched in biophysical as well as purely computational domains [1, 2]. It is understood that recurrent connections in the network can keep the information from distant past alive so that it can be recovered from the current state. The memory capacity of these networks are generally measured in terms of the accuracy of recovery of the past information [2-4]. Although the memory capacity strongly depends on the network's topology and sparsity [5-8], it can be significantly increased by exploiting any prior knowledge of the underlying structure of the encoded signal [9, 10]. Our approach to encoding memory stems from a focus on its utility for future prediction, rather than on the accuracy of recovering the past. In particular we are interested in encoding time varying signals from the natural world into memory so as to optimize future prediction. It is well known that most natural signals exhibit scale free long range correlations [11-13]. By exploiting this intrinsic structure underlying natural signals, prior work has shown that the predictive information contained in a finite sized memory system can be maximized if the past is encoded in a scale-invariantly coarse grained fashion [14]. Each node in such a memory system would represent a coarse grained average around a specific past moment, and the time window of coarse graining linearly scales with the past timescale.