SPECTra: Scalable Multi-Agent Reinforcement Learning with Permutation-Free Networks