Google Open-Sources Reformer Efficient Deep-Learning Model

Using a hashing trick for attention calculation and reversible residual layers, the Reformer can handle text sequences up to 1 million words while consuming only 16GB of memory on a single GPU accelerator. In a paper accepted for oral presentation at ICLR 2020, the team gave a detailed description of the model and several experiments. Since two of the major sources of large memory requirements of a standard Transformer model are the attention calculations and the network layer activations, the team applied space-saving techniques to each of these areas. To reduce the requirement for attention, Reformer uses approximate attention calculation via locality-sensitive hashing (LSH), reducing the memory requirements from \(O(N 2)\) to \(O(N\log{N})\), where N is the length of the input sequences. Using reversible layers reduces the need to store activations for every network layer; instead, only the last layer's activations need to be stored.