A Task details

A.1 Benchmark: autoassociative recall For the autoassociative memory benchmark task, we generate T memories, each of which is a uniformly randomly generated d-dimensional vector x During the storage phase, the key and value matrices are initialized to zero and each pattern is shown to the network sequentially with the network's global third factor q For storage, both the network's input and output layers are clamped to the input value x Figure A1: Autoassociative recall benchmark task. A.2 Beyond simple recall To test the network in a continual setting, rather than datasets of fixed length T, we use arbitrarily long datasets where the network is asked to recall a stimulus that was presented R timesteps ago. To ensure that the network is operating in steady state and therefore in the continual learning regime, we use a long trial duration T = max(1000, 20R). Figure A2a shows 30 timesteps of such a trial with R =2. Note that in a single trial, the delay interval between the stored stimulus and the query is always a fixed value R.