Generating Cool Storylines Using a T5 Transformer and Having Fun

The folks at Google AI published a paper "Exploring the Limits of Transfer Learning with a Unified Text-To-Text Transformer" and presented an empirical study on what type of pre-training approaches or transfer learning techniques work the best and then used that study to create a new model i.e. the Text-To-Text Transformer (T5). This transformer model was pre-trained on a much cleaner version of the Common Crawl Corpus and Google named it the Colossal Clean Crawled Corpus (C4). It worked out great also when you check out the model's ability and flexibility to be fine-tuned for a multitude of downstream NLP problems with very little to moderate data. To answer that question we would need to look at other Transformers like BERT, GPT, etc… All of them are pre-trained on a lot of data but to fine-tune on downstream tasks like classification a classification layer is added and outputs a class label or for NER it outputs a span of the input. But in T5 everything is Sequence2Sequence or like they say "Text-to-Text".