Scalable Multi-Agent Offline Reinforcement Learning and the Role of Information