spikedattention
Country:
- North America > United States > Texas > Travis County > Austin (0.04)
- Asia > South Korea > Daegu > Daegu (0.04)
Industry:
- Energy (0.49)
- Health & Medicine (0.46)
- Information Technology (0.46)
Technology:
Country:
- North America > United States > Texas > Travis County > Austin (0.04)
- Asia > South Korea > Daegu > Daegu (0.04)
Industry:
- Energy (0.49)
- Health & Medicine (0.46)
- Information Technology (0.46)
Technology:
SpikedAttention: Training-Free and Fully Spike-Driven Transformer-to-SNN Conversion with Winner-Oriented Spike Shift for Softmax Operation
Event-driven spiking neural networks(SNNs) are promising neural networks that reduce the energy consumption of continuously growing AI models. Recently, keeping pace with the development of transformers, transformer-based SNNs were presented. In this work, we propose a novel transformer-to-SNN conversion method that outputs an end-to-end spike-based transformer, named SpikedAttention. Our method directly converts the well-trained transformer without modifying its attention architecture. For the vision task, the proposed method converts Swin Transformer into an SNN without post-training or conversion-aware training, achieving state-of-the-art SNN accuracy on ImageNet dataset, i.e., 80.0\% with 28.7M parameters.
Technology: Information Technology > Artificial Intelligence > Machine Learning > Neural Networks (0.91)