From Centralized to Self-Supervised: Pursuing Realistic Multi-Agent Reinforcement Learning