Strengths: The paper offers two novel theoretical contributions. The first contribution is the introduction of the novel DA-MPI framework, which the authors prove is equivalent to the MD-MPI scheme (proposition 1). The second contribution is the derivation of two new performance bounds based on this framework, one for a purely KL-regularised objective and another for a KL and entropy regularised objective (theorems 1 and 2). These bounds have been missing from existing analysis and offer an important contribution to the NeurIPS community. The insight offered in the analysis of these bounds is valuable: we see that the first bound demonstrates that using KL-regularisation leads to a linear dependence on the horizon term, unlike the typical quadratic dependence for non regularised form.

TRPO, MPO, SAC, soft Q-learning, softmax DQN, DPP, etc) under one mirror descent framework and provides proofs for KL regularized value iteration. The paper shows that using KL regularization implicitly averages the estimates of the Q function, and using this result it shows a linear dependence of the approximation error of Q on the time horizon, whereas in many previous works with similar assumptions it was quadratic. This is a significant result. In addition, KL regularization ensures convergence in the case of independent and centered errors, which is not the case for standard approximate dynamic programming. The paper also examines how KL regularization interacts with entropy regularization, and presents empirical findings suggesting that KL regularization alone might be sufficient and better then entropy regularization, encouraging a lot of exploration in the beginning and less so as the policy deviates from being uniform.

Weaknesses: There were two primary weaknesses that I noticed in the paper: (1) The paper notes that the framework is different from prior curriculum learning work due to learning from prior learners, allowing it to be "data-driven rather than heuristic," but the consequences of that aren't explored. In particular, this may mean that many agents must be trained prior to getting to a good curricular policy, and this may be problematic for practitioners. This fact is somewhat hidden in the experimental evaluations because the evaluation of the optimized sequence of interventions doesn't include the performance of the first 30-100 learners. It would be helpful to either include a clearer argument for the importance of learning the curriculum policy over other approaches or to discuss the possible limitations of needing to learn from many learners (and perhaps the robustness claims would mitigate this limitation to some extent). I would also have liked to see results (perhaps in the appendix) for the sensitivity of the results to the number of learners prior to reaching an "optimized" point.

After the discussion all reviewers support acceptance, noting that the paper lays out a novel, clear, and general framework for safe online RL. This topic is very relevant to the NeurIPS community, and the paper should be disseminated. However, all reviewers expressed at least minor concerns. I strongly encourage the authors to consider this feedback so that they can improve the responses from future readers. In the discussion, it also became clear that a reviewer thought the paper described an agent interacting with *human* students - perhaps future clarifications can avoid this point of confusion.

Weaknesses: Below I list my concerns regarding the setup and reported results. In the finite case, devising an algorithm for the online setup posed more serious challenges than the generative setup. The restriction of the results to the generative setup hides the price to pay for the need to navigate in the MDP. Could you at least elaborate on explaining the potential difficulties and challenges involved in extending the results to the online case? Could one hope for a similar gain in the sample complexity (over structure-oblivious algorithms)?

The contributions were unanimously appreciated (the paper introduces an interesting structure, the regret analysis including the low-matrix estimation part is interesting). We recommend the paper for acceptance and encourage the authors to account for the reviewers' comments when preparing the camera-ready version of the paper.

The paper formulates the problem of optimizing a strategy for crawling remote contents to track their changes as an optimization problem called the freshness crawl scheduling problem. This problem is an obviously important problem in applications like Internet search engine, and the presented formulation seems to give a practical solution to those applications. The paper presents an algorithm for solving the freshness crawl scheduling problem to optimality, assuming that the contents change rates are known. The idea behind the algorithm is based on the deep understanding of statistics and continuous optimization, and it seems to me that the contribution is solid (although I could not very all the technical details). For the case where the contents change rates are not known, a reinforcement learning algorithm is presented.

This paper presents an RL-approach for optimizing web crawling strategy by modeling freshness of the remote content. Reviewers were unanimously in favor of accepting the paper, appreciating the formulation of the problem and the extent of the scale of experiments.

Reviewers agree that this is a borderline paper, but overall are happy with the rebuttal and have adjusted scores slightly. There is also agreement that the paper is well-written and clear, with supported contribution, but with somehow minor algorithmic improvements. Reviewers seem ok to accept if the authors provide additional clarification in their crc as provided in the rebuttal. As an AC I am in favor of acceptance.

Additional Feedback: --------------- Post Rebuttal ------------------ Comments on the rebuttal: - The authors provide results of fusing existing SOTA trackers with the proposed switching strategy. On all datasets, results are marginally better than selecting the best of the two trackers (0.001 - 0.007 AUC/EAO). This improvement is rather minor, and difficult to put into context since other fusion methods are not compared. There are naive baselines for doing this, for example simply averaging the two bounding boxes. But exactly this topic has attracted substantial research interest over the years: MCCT, LCT, MEEM, [N.