Figure 1A illustrates this trade-off for a single represented history. We note that dynamics amplifying the input will in general also amplify the noise. Trade-offs in a noisy, constrained LDS. All three vectors are assumed to have unit length. The increased precision is thus enabled by discarding irrelevant information.

This is a nice contribution in that it combines several different approaches (efficient coding, neuroscience/neural modeling, MDPs) in a conceptually novel way (R1, R4, R5), with R4 commenting that it's likely to be of great impact to the wider community. On the other hand, R3 saw limited conceptual novelty and believes that some prior work on policy compression has been understated. In general, I'm inclined to agree with other reviewers that it's fairly well-positioned with regard to prior work (R1). R4 praised the clarity of the writing, and other reviewers didn't have any issues with the presentation. R5 expressed concern that the results are mainly qualitative, and not particularly novel, despite the novelty of the approach itself.

Weaknesses: One main problem is that the paper does not contain a plausible method for learning. Not only would this likely be extremely hard (for the informational measures), but there could also be a complex interaction between things like compression, exploration and learning. Although it is certainly interesting to think about the difference between model-based and model-free representations, I wasn't completely convinced by the arguments in the paper. If I understand correctly, the habitual agent would have a partly open-loop character to it (ie it would ignore parts of the observation) - this is dangerous in anything but a completely stationary world; and since animals seem to continue to possess their model-based methods even after control has become habitized, it would also seem that the suggestion would be that animals would maintain two separate representations, one MB and the other MF, which seems wasteful. The experiments could also have been more convincing.

Higher-order brain areas such as the frontal cortices are considered essential for the flexible solution of tasks. However, the precise computational role of these areas is still debated. Indeed, even for the simplest of tasks, we cannot really explain how the measured brain activity, which evolves over time in complicated ways, relates to the task structure. Here, we follow a normative approach, based on integrating the principle of efficient coding with the framework of Markov decision processes (MDP). More specifically, we focus on MDPs whose state is based on action-observation histories, and we show how to compress the state space such that unnecessary redundancy is eliminated, while task-relevant information is preserved.