Towards Efficient Collaboration via Graph Modeling in Reinforcement Learning