Review for NeurIPS paper: The LoCA Regret: A Consistent Metric to Evaluate Model-Based Behavior in Reinforcement Learning

This paper proposes a method for identifying model-based behavior in RL agents (the "LoCA regret"), which can be used without knowing anything about the internal structure of the agent itself. This method is demonstrated to correctly distinguish between classical known model-free and model-based agents. It is also used to analyze MuZero, revealing that although MuZero is in principle a model-based algorithm, it does not make optimal use of its model. The reviewers agreed that the LoCA regret is a useful metric, and felt that doing careful evaluation of agents by designing metrics like this is an important area of research in RL. I agree, and found very interesting the demonstration that just because a particular algorithm makes use of a model, doesn't necessarily mean that the algorithm will have the properties that we think of as being associated with model-based algorithms. While there was some debate during the discussion period about some of the choices regarding the calculation of the LoCA regret (e.g.