Pre-training BERT from scratch with cloud TPU

In this experiment, we will be pre-training a state-of-the-art Natural Language Understanding model BERT on arbitrary text data using Google Cloud infrastructure. With this guide, you will be able to train a BERT model on arbitrary text data. This is useful if a pre-trained model for your language or use case is not available in open source. This guide is intended for NLP researchers who are excited with the BERT technology but are not satisfied with the performance of the available open-sourced models. For persistent storage of training data and model, you will require a Google Cloud Storage bucket.