We present a novel self-supervised approach for hierarchical representation learning and segmentation of perceptual inputs in a streaming fashion.

Large pretrained models can be used as annotators, helping replace or augment crowdworkers and enabling distilling generalist models into smaller specialist models.

Several recent works demonstrate that transformers can implement algorithms like gradient descent. By a careful construction of weights, these works show that multiple layers of transformers are expressive enough to simulate iterations of gradient descent.

This restricted model may be thought of as "side-information", and can come

The training of language models is typically not an end-to-end process.

Linear RNN architectures, like Mamba, can be competitive with Transformer models in language modeling while having advantageous deployment characteristics. Given the focus on training large-scale Transformer models, we consider the challenge of converting these pretrained models for deployment.

First-order optimization methods are a foundational tool in machine learning.