InhibiDistilbert: Knowledge Distillation for a ReLU and Addition-based Transformer

Transformer-based language models have revolutionized natural language processing (NLP), achieving state-of-the-art performance across a wide range of tasks, from machine translation to sentiment analysis [6]. However, the computational and energy demands of these models, particularly those arising from the self-attention mechanism, pose significant challenges for deployment in resourceconstrained environments. Although highly effective, the self-attention mechanism relies heavily on matrix multiplications, which are computationally expensive and energy-intensive As the scale of transformer models continues to grow, so does their environmental impact, with studies estimating that training a single large model can emit as much carbon as five cars over their lifetimes [5]. This has spurred research into more efficient alternatives, including model compression techniques such as knowledge distillation [3] and alternative attention mechanisms, like ReLUFormer [4] or Linformer [8]. Another alternative is the inhibitor attention [2], which was introduced as a means to avoid using the softmax function and matrix multiplications.