Understanding how BERT reasons

BERT is now the go-to model framework for NLP tasks in industry, in about a year after it was published by Google AI. When released, it achieved state-of-the-art results on a variety of NLP benchmarks. It's referred to as a framework because BERT is not a model per se, but in the words of the authors themselves, it is a "method of pre-training language representations, meaning that we train a general-purpose "language understanding" model on a large text corpus (like Wikipedia), and then use that model for downstream NLP tasks that we care about (like question answering)." For the purpose of this blogpost, when we refer to a BERT model, we mean a model based on the BERT architecture and fine tuned for a particular task using pre-trained weights. Several papers have attempted to explain it, and created a field that the people at HuggingFace call Bertology .