Boosting classification reliability of NLP transformer models in the long run

Introduction A key goal of machine learning projects is some form of classification of the input data. This classification is typically done so that both the training and the data to be classified come from the same period and data set. In practice, however, it may often be the case that a particular classification is extended to a different set of data and/or to a different period. The need and the possibility to extend classification over time is strongly supported by increasing digitization as updated datasets are more frequently available for industrial and scientific research purposes. But how long is a classification suitable for, and when is it worth re-training our model? Since the much-quoted Google Flu case, every researcher using machine learning knows that models should not be blindly trusted and that it is crucial to revise them in time (Lazer et al., 2014). The need for retraining may be particularly relevant in cases where the domain under study changes rapidly over time. In the various natural language processing (NLP) projects, this problem is common, as language and language use can change within a short period of time on a given topic (Kulkarni et al., 2015). Furthermore, neural network-based black-box models complicate the problem, as they offer little insight into how a particular classification model works, making it harder to identify what changes in content or context might cause the model to break down. This black-box feature is a problem with the transformer-based NLP models currently used for classification tasks in data-mining projects. Transformer-based machine learning models have become an important tool for many natural language processing (NLP) tasks since the introduction of the method (Vaswani et al., 2017).