Q-Learning-Driven Adaptive Rewiring for Cooperative Control in Heterogeneous Networks

Cooperation emergence in multi-agent systems represents a fundamental statistical physics problem where microscopic learning rules drive macroscopic collective behavior transitions. We propose a Q-learning-based variant of adaptive rewiring that builds on mechanisms studied in the literature. This method combines temporal difference learning with network restructuring so that agents can optimize strategies and social connections based on interaction histories. Through neighbor-specific Q-learning, agents develop sophisticated partnership management strategies that enable cooperator cluster formation, creating spatial separation between cooperative and defective regions. Using power-law networks that reflect real-world heterogeneous connectivity patterns, we evaluate emergent behaviors under varying rewiring constraint levels, revealing distinct cooperation patterns across parameter space rather than sharp thermodynamic transitions. Our systematic analysis identifies three behavioral regimes: a permissive regime (low constraints) enabling rapid cooperative cluster formation, an intermediate regime with sensitive dependence on dilemma strength, and a patient regime (high constraints) where strategic accumulation gradually optimizes network structure. Comparative analysis against Bush-Mosteller stimulus-response learning demonstrates that Q-learning's temporal credit assignment capabilities produce superior cooperation outcomes, particularly under intermediate rewiring constraints where long-term relationship assessment becomes crucial. Quantitative analysis reveals that increased rewiring frequency drives large-scale cluster formation with power-law size distributions. Our results establish a new paradigm for understanding intelligence-driven cooperation pattern formation in complex adaptive systems, revealing how machine learning serves as an alternative driving force for spontaneous organization in multi-agent networks. Introduction Ensuring cooperative control in distributed engineered systems and applications is a daunting challenge across diverse domains. In distributed resource management, cooperative agents must dynamically adapt to balance local demands and maintain global performance [1]; in urban traffic networks, intersections must exchange information to optimize flows [2, 3]; in robotic swarms, unmanned aerial vehicles or mobile robots must align actions for collective tasks under uncertainty [4, 5]. Apparently, in each case, the performance of the overall system, including throughput, latency, reliability, and safety, depends on the ability of autonomous agents to adapt strategies and restructure interactions in dynamic environments. Enhancing cooperation among agents is therefore essential, since insufficient coordination can lead to cascading failures, degraded performance, or even systemic collapse in critical infrastructures.