Masked Particle Modeling on Sets: Towards Self-Supervised High Energy Physics Foundation Models

These models also represent a scale in both model size and data size that have not been addressed in HEP. In this work, we aim to take the first steps towards building such While Artificial Intelligence (AI) and Machine Learning a HEP foundation model, focusing on developing HEP (ML) are already playing a major role in the analysis of data specific SSL strategies, whilst keeping an eye on how high energy physics (HEP) data, the HEP community well such strategies may scale in the future. We propose a has yet to benefit from the self-supervised learning (SSL) masked particle modeling (MPM) scheme, akin to masked based approaches to building large foundation models language modeling (MLM) in NLP, for self-supervised (FM) [1] that have been pioneered in natural language learning on unlabeled data consisting of sets of particles processing (NLP) [2-5] and computer vision (CV) [6-8]. in a collider physics environment. In doing so, we propose These modern approaches use SSL to pre-train models a novel scheme to apply masked modeling strategies to on vast data sets in order to learn generic representations unordered sets of inputs. of the data. Such models can then be efficiently finetuned with small datasets for a variety of downstream This work aims to generalize the language-inspired tasks. The self-supervised pre-training of a FM produces MLM-type training scheme to HEP scientific data. The a model that is also referred to as the "backbone", as it paradigm involves extracting semantic meaning and understanding can serve as the information extraction component for of the whole by predicting the missing (masked) downstream models. This concept significantly expands pieces, referred to as tokens, thereby considering the collective the possibilities for learning robust and meaningful data impact of individual input elements.