Nevertheless, it remains elusive that whether the Transformer-based architecture can do molecular modeling as good as equivariant GNNs.

Spiking neural networks (SNNs) represent a promising approach to developing artificial neural networks that are both energy-efficient and biologically plausible.

Positional encoding plays a crucial role in transformers, significantly impactingmodel performance andlength generalization. Prior research hasintroduced absolute positional encoding (APE) and relative positional encoding (RPE) to distinguish token positions in given sequences.

Second, on channel aspect, representation exhibits diversity ondifferent channels. But the scarce data can not enable ViTs to learn strong enough representation for accurate recognition.