Goto

Collaborating Authors

 Government



Symmetry-Informed Governing Equation Discovery

Neural Information Processing Systems

Despite the advancements in learning governing differential equations from observations of dynamical systems, data-driven methods are often unaware of fundamental physical laws, such as frame invariance.



Faster Neighborhood Attention: Reducing the O (n 2) Cost of Self Attention at the Threadblock Level

Neural Information Processing Systems

Neighborhood attention reduces the cost of self attention by restricting each token's attention span to its nearest neighbors. This restriction, parameterized by a window size and dilation factor, draws a spectrum of possible attention patterns between linear projection and self attention. Neighborhood attention, and more generally sliding window attention patterns, have long been bounded by infrastructure, particularly in higher-rank spaces (2-D and 3-D), calling for the development of custom kernels, which have been limited in either functionality, or performance, if not both.