Details and more visualizations are in Appendices C and D.

Transformers, renowned for their self-attention mechanism, have achieved state-of-the-art performance across various tasks in natural language processing, computer vision, time-series modeling, etc. However, one of the challenges with deep Transformer models is the oversmoothing problem, where representations across layers converge to indistinguishable values, leading to significant performance degradation. We interpret the original self-attention as a simple graph filter and redesign it from a graph signal processing (GSP) perspective. We propose graph-filter-based self-attention (GFSA) to learn a general yet effective one, whose complexity, however, is slightly larger than that of the original self-attention mechanism. We demonstrate that GFSA improves the performance of Transformers in various fields, including computer vision, natural language processing, graph pattern classification, speech recognition, and code classification.

Tech giant Google on Monday announced to invite applications for the seventh batch of its startup accelerator programme in India that aims to nurture startups looking to leverage the potential of Artificial Intelligence (AI) and Machine Learning (ML). Google for Startups Accelerator (GFSA) is a three-month, equity-free accelerator programme for Seed to Series A tech startups in India. It has, so far, helped to accelerate the growth journeys of over 130 startups in India. For the next batch Google is looking to support startups that are focused on utilising the transformational technologies AI and ML. "We are looking for Indian startups working across different sectors using AI and ML to solve complex problems and tackle the toughest challenges in their industry," the tech giant said in a statement.