Domain-Specific Language Model Post-Training for Indonesian Financial NLP

One of the notable examples Recently, self-supervised pre-training of contextual language is Bidirectional Encoder Representations from Transformers models on large general domain corpora, such as ELMo (BERT), which has become a standard benchmark for training [7], ULM-Fit [8], XLNet [9], GPT [10], BERT [2], and NLP models for various downstream tasks. Another example is IndoBERT [1] have significantly improved performance on IndoBERT, the implementation of BERT specific for Indonesian various natural language processing downstream tasks, including language which also performs well as a building block sentence classification, token classification, and question for training task-specific NLP models for Indonesian language answering. IndoBERT, as the foundation of this research, is an [1]. However, those pre-training works focus on the general implementation of BERT in Indonesian language. IndoBERT domain in which the unlabeled text data are collected from has similar model architecture as BERT in which it is a Web domains, newswire, Wikipedia, and BookCorpus [1], [2].