Is BERT Always the Better Cheaper Faster Answer in NLP? Apparently Not.

Summary: Since BERT NLP models were first introduced by Google in 2018 they have become the go-to choice. New evidence however shows that LSTM models may widely outperform BERT meaning you may need to evaluate both approaches for your NLP project. Over the last year or two, if you needed to bring in an NLP project quickly and with SOTA (state of the art) performance, increasingly you reached for a pretrained BERT module as the starting point. Recently however there is growing evidence that BERT may not always give the best performance. In their recently released arXiv paper, Victor Makarenkov and Lior Rokach of Ben-Gurion University share the results of their controlled experiment contrasting transfer-based BERT models with from scratch LSTM models.