A Look at IGLOO: Slicing the Features Space to Represent Sequences – IAM Network

Sequences are most important in deep learning. Whether it is in natural language processing (NLP) or for biological data (RNA sequences), neural networks try to find a representation for sequences of tokens and classify them accordingly or generate new ones following a given logic. There are generally two approaches for this task: The first one is Recurrent Neural Networks (RNN) and its variants (GRU and LSTM), the second one is Transformers. The first method looks at elements in the sequence recursively while the second one focuses on self-attention between elements of the sequence.Each approach has had great success but neither is particularly suited for long sequences. Experiments show that LSTM have a difficult time dealing with sequences longer than 5000 steps, while Transformers are not adapted to it because of the large memory requirements.