Review for NeurIPS paper: Cooperative Heterogeneous Deep Reinforcement Learning

The exact mechanic of the policy transfer between different algorithm is not given. Given the content, I may assume that "transfer" means a simple copying of the parameters, but I remain unsure. When augmenting the experience buffer with other algorithm, it would be nice to clarify why it does (not) introduce any bias in the data. It seems that the different parts of the framework could be replaced by a different way of "tinkering" with a algorithm or its hyperparameters. E.g., the auxiliary on-policy algorithms are here mainly for exploration, but the exploration of the main off-policy algorithm itself can be easily controlled and I suspect it can, with the right setting, work as good as the given complicated framework. The global and local experience buffer seems more like a hack.