Generative Pretraining at Scale: Transformer-Based Encoding of Transactional Behavior for Fraud Detection

In the dynamic world of digital finance, protecting transactions from fraud is an increasingly intricate challenge. Machine learning has advanced behavioral analysis, yet it faces difficulties with the detailed patterns of transaction data--both vast in volume and complex in nature. Conventional models fall short in decoding this data accurately. Our study introduces a novel method that applies Generative Pretrained Transformers, acclaimed for language understanding, to model financial transactions. By pretraining on extensive user payment data, our model overcomes the common obstacles of behavioral sequence analysis such as the need for large amounts of labeled data. We detail three major contributions: firstly, an innovative autoregressive pretraining method designed specifically for financial data. Secondly, we introduce a differential convolution structure that enhances the model's capacity for anomaly detection. Lastly, from a methodological perspective, we demonstrate the model's flexibility in adapting to diverse financial scenarios, and from an application standpoint, we validate its scalability through extensive testing in our dataset. Our work not only moves the needle forward in financial fraud detection but also showcases the potential of unsupervised learning in environments where protecting data and user privacy is crucial.