RITA: a Study on Scaling Up Generative Protein Sequence Models

Downstream open-source experimentation is important to discover surprising and unpredictable capabilities In this work we introduce RITA: a suite of autoregressive that are hard to discern without large-scale experimentation generative models for protein sequences, (Ganguli et al., 2022). This was recently exemplified with up to 1.2 billion parameters, trained on over when independent researchers discovered that AlphaFold 2 280 million protein sequences belonging to the (Jumper et al., 2021) could successfully predict multimer UniRef-100 database. Such generative models interactions, even though it had only been trained to predict hold the promise of greatly accelerating protein the structure of single protein chains (Yoshitaka, 2021; Baek, design. We conduct the first systematic study of 2021). In addition, there exists no systematic study about how capabilities evolve with model size for autoregressive the evolution of capabilities with respect to model size in transformers in the protein domain: the protein domain: Rao et al. (2020) and Rives et al. (2021) we evaluate RITA models in next amino acid prediction, provided such a study for bidirectional transformers, and zero-shot fitness, and enzyme function Madani et al. (2020) simply noted that their largest model prediction, showing benefits from increased scale.