GPT-2: 6-Month Follow-Up

We're releasing the 774 million parameter GPT-2 language model after the release of our small 124M model in February, staged release of our medium 355M model in May, and subsequent research with partners and the AI community into the model's potential for misuse and societal benefit. We're also releasing an open-source legal agreement to make it easier for organizations to initiate model-sharing partnerships with each other, and are publishing a technical report about our experience in coordinating with the wider AI research community on publication norms. To date, there hasn't been a public release of a 1558M parameter language model, though multiple organizations have developed the systems to train them, or have publicly discussed how to train larger models. For example, teams from both NLP developer Hugging Face and the Allen Institute for Artificial Intelligence (AI2) with the University of Washington have explicitly adopted similar staged release approaches to us. Since February, we've spoken with more than five groups who have replicated GPT-2[1].