AttentionDrop: A Novel Regularization Method for Transformer Models

Transformer architectures [1] leverage multi-head self-attention to capture long-range dependencies, leading to breakthroughs in NLP, Computer Vision, and beyond. Despite their success, large-scale transformers with billions of parameters are prone to overfitting when data is scarce or noisy. Traditional regularization methods such as Dropout [2] and weight decay [52] target network weights or activations but do not directly address the attention mechanism, which lies at the core of transformer expressivity. This work is based on the hypothesis of that overly sharp attention distributions, where a few tokens dominate the context, can cause brittle representations. By injecting controlled stochastic perturbations into the attention logits or weights during training, the model can be encouraged to explore alternative context paths, thereby improving robustness and generalization. Following are the contributions of this research: Introducing AttentionDrop, the first family of regularizers that directly perturb self-attention distributions during training. Following three variants have been formalized: 1. Hard Attention Masking: to randomly zero out top-k attention logits per query to encourage diverse context utilization.