TransformLLM: Adapting Large Language Models via LLM-Transformed Reading Comprehension Text

Large Language Models (LLM) domain-adaptive pre-training, also known as continued pre-training on domainspecific corpora [12], is a technique that has been proven effective in adapting large language models (LLMs) to specific domains [35, 5]. This approach allows LLMs to leverage their general language understanding capabilities while incorporating domain-specific knowledge, which can benefit downstream domain-specific tasks at reduced costs [22, 26, 27]. In this process, the LLM is further pre-trained using raw data from the specific domain, such as biomedicine, finance, or law. This helps the LLM gain domain knowledge, which is demonstrated by its improved performance in fine-tuning and knowledge probing evaluations within those domains [20, 1, 2]. However, a notable drawback is that continued pre-training on raw domain corpora can lead to a significant drop in the LLM's prompting performance, potentially due to the specialized nature of the domain-specific data [11]. Despite this trade-off, domain-adaptive pre-training remains a promising approach for adapting LLMs to specific domains, capitalizing on their general language understanding capabilities while tailoring them to domain-specific tasks and knowledge. Ongoing research efforts aim to mitigate the potential negative impacts on prompting performance while maximizing the benefits of domain-specific knowledge acquisition [10, 28]. The notion of reading comprehension was suggested in [6], where instead of continuing to train a large language model on domain-specific raw data, the raw texts be converted into reading comprehension materials. In this approach, each text is followed by related tasks, transitioning the model from a "reading" phase to a "comprehension" phase.