Transformer-based models have recently become wildly successful across a diverse set of domains. At the same time, recent work has shown that Transformers are inherently low-pass filters that gradually oversmooth the inputs, reducing the expressivity of their representations. A natural question is: How can Transformers achieve these successes given this shortcoming? In this work we show that in fact Transformers are not inherently low-pass filters. Instead, whether Transformers oversmooth or not depends on the eigenspectrum of their update equations. Our analysis extends prior work in oversmoothing and in the closely-related phenomenon of rank collapse. We show that many successful Transformer models have attention and weights which satisfy conditions that avoid oversmoothing. Based on this analysis, we derive a simple way to parameterize the weights of the Transformer update equations that allows for control over its spectrum, ensuring that oversmoothing does not occur. Compared to a recent solution for oversmoothing, our approach improves generalization, even when training with more layers, fewer datapoints, and data that is corrupted.

Artificial intelligence (AI) far too often pops up as a term used vaguely to refer to any process that appears to involve more computers than it did twenty years ago. But concrete examples of how this informatics technology can improve fields like life sciences are harder to come by. I recently spoke to Krishnan Nandabalan, founder, CEO and president of InveniAI, which aims to use AI techniques to more quickly identify pharmacological compounds and get drugs to patients faster, to get the full picture. Ruairi Mackenzie (RM): How would you like to set the record straight about AI in the life sciences? Krishnan Nandabalan (KN): AI is used as a buzzword now.