Google's AI language model Reformer can process the entirety of novels

Whether it's language, music, speech, or video, sequential data isn't easy for AI and machine learning models to comprehend -- particularly when it depends on extensive surrounding context. For instance, if a person or an object disappears from view in a video only to reappear much later, many algorithms will forget how it looked. Researchers at Google set out to solve this with Transformer, an architecture that extended to thousand of words, dramatically improving performance in tasks like song composition, image synthesis, sentence-by-sentence text translation, and document summarization. But Transformer isn't perfect by any stretch -- extending it to larger contexts makes apparent its limitations. Applications that use large windows have memory requirements ranging from gigabytes to terabytes in size, meaning models can only ingest a few paragraphs of text or generate short pieces of music.