In-Context Learning for Few-Shot Molecular Property Prediction

In-context learning has become an important approach for few-shot learning in Large Language Models because of its ability to rapidly adapt to new tasks without fine-tuning model parameters. However, it is restricted to applications in natural language and inapplicable to other domains. In this paper, we adapt the concepts underpinning in-context learning to develop a new algorithm for few-shot molecular property prediction. Our approach learns to predict molecular properties from a context of (molecule, property measurement) pairs and rapidly adapts to new properties without fine-tuning. On the FS-Mol and BACE molecular property prediction benchmarks, we find this method surpasses the performance of recent meta-learning algorithms at small support sizes and is competitive with the best methods at large support sizes. In-context learning describes an emergent property of large language models (LLMs) that enables them to solve new tasks from only a few demonstrations and without any gradient updates to the model parameters (Brown et al., 2020). This capacity to rapidly adapt to new tasks contrasts sharply with typical few-shot learning algorithms that either use gradient updates, or distance computations to prototypical class centroids, to adapt the pre-trained model to the few-shot learning objective. As a result, in-context learning has become a powerful approach for few-shot learning applications in natural language; however, it is inapplicable to other domains as it uses a language modeling objective to train the model. One such domain is molecular science where few-shot learning is critical to drug discovery. After a biological target has been identified, finding small molecules that inhibit this target may lead to desirable outcomes. For example, inhibiting the protein 15-PGDH with a small molecule inhibitor leads to rejuvenation of aged skeletal muscle tissue in animal studies, effectively reverse-aging the cells (Palla et al., 2021).