Recently, recurrent models based on linear state space models (SSMs) have shown promising performance in language modeling (LM), competititve with transformers. However, there is little understanding of the in-principle abilities of such models, which could provide useful guidance to the search for better LM architectures. We present a comprehensive theoretical study of the capacity of such SSMs as it compares to that of transformers and traditional RNNs. We find that SSMs and transformers have overlapping but distinct strengths. In star-free state tracking, SSMs implement length-generalizing solutions to problems that transformers struggle to represent exactly. They can also model bounded hierarchical structure with optimal memory even without simulating a stack. On the other hand, we identify a design choice in current SSMs that limits their expressive power.

Deep learning on tabular data, and particularly tabular representation learning, has recently gained growing interest. However, representation learning for relational databases with multiple tables is still an underexplored area, which may be attributed to the lack of openly available resources.

Recent strides in molecular design have introduced advanced generative models that excel in de-novo generation and optimization within 3D molecular space (Isert et al. [2023]).

Most currently deployed LLMs undergo continuous training or additional finetun-ing. By contrast, most research into LLMs' internal mechanisms focuses on models

The rapid development of large language and vision models (LL VMs) has been driven by advances in visual instruction tuning.

The architecture of Vision Transformers (ViTs), particularly the Multi-head Attention (MHA) mechanism, imposes substantial hardware demands.

Pre-training plays a vital role in various vision tasks, such as object recognition and detection.