I Fine-Tuned GPT-2 on 100K Scientific Papers

After fine-tuning the model, I wanted to understand what the model has learned and how the generated text is influenced by the fact that paper abstracts were used for training. First, I generated a sample text by using "the role of recommender systems" as a prompt. This result sounded somehow copied & pasted from one of the existing abstracts, but after a check with some anti-plagiarism solutions, I realized that it is 100% unique. During learning, the model captured common features of the abstracts and learned how to replicate them while still generating fresh text. Interestingly, the model used scientific language and common expressions: The previous works…, In this paper…, We propose…, The experimental result….