Low-Complexity LSTM Training and Inference with FloatSD8 Weight Representation

--The FloatSD technology has been shown to have excellent performance on low-complexity convolutional neural networks (CNNs) training and inference. In this paper, we applied FloatSD to recurrent neural networks (RNNs), specifically long short-term memory (LSTM). Moreover, the arithmetic precision for accumulations and the master copy of weights were reduced from 32 bits to 16 bits. We demonstrated that the proposed training scheme can successfully train several LSTM models from scratch, while fully preserving model accuracy. Finally, to verify the proposed method's advantage in implementation, we designed an LSTM neuron circuit and showed that it achieved significantly reduced die area and power consumption. I NTRODUCTION Recently, a great many studies have worked on proposing low-complexity, high-performance neural networks (NN) training and inference methods as well as their hardware architectures. The main objective is to reduce the huge computational and memory storage/access needed by popular modern NN models.