Multi-agent reinforcement learning has recently shown great promise as an approach to networked system control. Arguably, one of the most difficult and important tasks for which large scale networked system control is applicable is common-pool resource management. Crucial common-pool resources include arable land, fresh water, wetlands, wildlife, fish stock, forests and the atmosphere, of which proper management is related to some of society's greatest challenges such as food security, inequality and climate change. Here we take inspiration from a recent research program investigating the game-theoretic incentives of humans in social dilemma situations such as the well-known \textit{tragedy of the commons}. However, instead of focusing on biologically evolved human-like agents, our concern is rather to better understand the learning and operating behaviour of engineered networked systems comprising general-purpose reinforcement learning agents, subject only to nonbiological constraints such as memory, computation and communication bandwidth. Harnessing tools from empirical game-theoretic analysis, we analyse the differences in resulting solution concepts that stem from employing different information structures in the design of networked multi-agent systems. These information structures pertain to the type of information shared between agents as well as the employed communication protocol and network topology. Our analysis contributes new insights into the consequences associated with certain design choices and provides an additional dimension of comparison between systems beyond efficiency, robustness, scalability and mean control performance.

Multi-agent reinforcement learning has recently shown great promise as an approach to networked system control. Arguably, one of the most difficult and important tasks for which large scale networked system control is applicable is common-pool resource management.

Weaknesses: - In multi-agent reinforcement learning research, Schelling diagrams are normally plotted as a function of the number of *other cooperators* (besides the focal agent making the decision), i.e. C - 1, rather than the total number of cooperators, C, as was done here. Either way is certainly correct in principle, Schelling said as much in the original 1973 paper. However, there are several reasons why the C - 1 parameterization is convenient. For instance, it lets you read off game theoretic properties from the diagram more easily. To see if cooperation or defection is favored for a particular number of other cooperators, you simply compare a point on the R_c curve to the point on the R_d curve that is right above it.

The paper is modelling MARL problems under the angle of social dilemma, and tries to tackle the problem of common-pool resource management. The authors do not introduce a novel method, instead this paper is a comparison of a wide range of existing relevant algorithms on a single problem (water management). The experiments are well motivated and in general, the paper is very clear. My understanding is that although the paper focuses on a water management, it is aimed as a more general survey of the quality of current MARL algorithms on common-pool resource management. The authors argue that water management is a good example to study because it is critical and life-supporting, and safety issues are very relevant.

Multi-agent reinforcement learning has recently shown great promise as an approach to networked system control. Arguably, one of the most difficult and important tasks for which large scale networked system control is applicable is common-pool resource management. Crucial common-pool resources include arable land, fresh water, wetlands, wildlife, fish stock, forests and the atmosphere, of which proper management is related to some of society's greatest challenges such as food security, inequality and climate change. Here we take inspiration from a recent research program investigating the game-theoretic incentives of humans in social dilemma situations such as the well-known \textit{tragedy of the commons}. However, instead of focusing on biologically evolved human-like agents, our concern is rather to better understand the learning and operating behaviour of engineered networked systems comprising general-purpose reinforcement learning agents, subject only to nonbiological constraints such as memory, computation and communication bandwidth.