Similar to the Markov decision process (MDP) framework in discrete-time, the continuous-time controlled diffusion processes provide a natural framework for modeling such continuous-time decision-making problems. With discrete-time observations from the continuous-time dynamics, the continuous-time RL problem can be viewed as a discrete-time MDP, allowing us to apply standard techniques. In particular, the model-free RL algorithms offers flexibility of function approximation. By fitting the value function and/or control policy with powerful statistical learning models including neural networks, one can efficiently learn the optimal decisions in high-dimensional and complex environments. Despite the empirical success, however, the theoretical understanding of continuous-time RL algorithms is still in its infancy. In particular, when applied to continuous-time diffusion processes, the statistical guarantees for value learning algorithms are largely unknown. The theoretical gap also leads to practical limitations, as the fundamental tradeoffs in the choice of function approximations, discretization step length, and the trajectory length remain elusive. In this work, we aim to bridge this gap by providing sharp statistical guarantees for value function estimation in continuous-time diffusion processes.

Data preprocessing is a critical yet frequently neglected aspect of machine learning, often paid little attention despite its potentially significant impact on model performance. While automated machine learning pipelines are starting to recognize and integrate data preprocessing into their solutions for classification and regression tasks, this integration is lacking for more specialized tasks like survival or time-to-event models. As a result, survival analysis not only faces the general challenges of data preprocessing but also suffers from the lack of tailored, automated solutions in this area. To address this gap, this paper presents 'CleanSurvival', a reinforcement-learning-based solution for optimizing preprocessing pipelines, extended specifically for survival analysis. The framework can handle continuous and categorical variables, using Q-learning to select which combination of data imputation, outlier detection and feature extraction techniques achieves optimal performance for a Cox, random forest, neural network or user-supplied time-to-event model. The package is available on GitHub: https://github.com/datasciapps/CleanSurvival Experimental benchmarks on real-world datasets show that the Q-learning-based data preprocessing results in superior predictive performance to standard approaches, finding such a model up to 10 times faster than undirected random grid search. Furthermore, a simulation study demonstrates the effectiveness in different types and levels of missingness and noise in the data.

The authors present two losses for improving emergent communication, addressing concerns laid out in previous work (Lowe 2019). One is the concern that the speaker agent may be communicating generic messages and not ones relevant to the particular sitation. A loss here encourages the agent to send messages that are correlated with their observation, based on maximizing mutual information between them. A second concern is that the listener agent may not be conditioning their behavior on the communication, and in this case an extra loss Both constraints are intuitive, and phrasing them as losses doesn't seem to be particularly challenging. However, as with many issues in emergent communication, such a judgement may gloss over hidden difficulties in a complex optimization problem, and this appears to be the case here, requiring some non-obvious sidestepping to provide losses with better convergence.

This paper proposes simple but potentially quite significant ways to encourage useful emergent communication in MARL. The paper is well written with a nice empirical analysis. The reviewers had some concerns about the coverage of related work and the empirical comparison to Jacques et al. but it seems easily feasible to address these in the final version. The authors are strongly encouraged to add to the final version a comparison to Jacques et al. in Treasure Hunt, regardless of whether the results favour the proposed method.

Additional Feedback: Could you provide a discussion on the main difficulties of analysing double Q-learning in the function approximation setting? The discussion on the exploration policy in the asynchronous version is confusing: Assumption 1 (existance of the covering number L) suggests a deterministic exploration policy, yet there is a claim on line 262 that L S A; why not? Can Assumption 1 is replaced by its "with high probability" equivalent? In Step I of Part I of the proof of Theorem 1 it would really help if you can provide an intuition on why z_t still a martingale difference sequence although you are dealing with interconnected stochastic processes. Minor comments: - R_t is defined in line 121 and then again in line 136; you may want to merge the two definitions and state in line 121 that t denotes time.

The reviewers agreed that this paper provided a nice analysis of double Q-learning, thereby filling an open gap in the theoretical understanding of such algorithms, and unanimously recommended acceptance.

Summary: The paper investigates the usefulness of modeling human behavior in human-ai collaborative tasks. In order to study this question, the paper introduces an experimental framework that consists of: a) modeling human behavior using imitation learning, b) training RL agents in several modes (self-play, trained agains human imitator, etc.), c) measuring the joint performance of human-AI collaboration. Using both simulation based experiments and a user study the paper showcases the importance of accounting for human behavior in designing collaborative RL agents. Comments: The topic of the paper is interesting and important for modern hybrid human-AI decision making systems. This seems like a well written paper with solid contributions: to the best of my knowledge, no prior work has systematically investigated the utility of human modeling in the context of human-AI collaboration in RL.

Weaknesses: My major concerns: 1. line 248 suggested linear programming could be used in ConPlanner, but instead the experiment tested on different unconstrained RL planners under Lagrangian heuristic. I think the papers should have compared results of different constrained problem solver. While theoretical proof was plenty, the paper didn't provide any empirical support, making this method less intuitive. Although the paper claimed they compared the proposed framework with other concave-convex approaches, the problems they experimented on didn't seem to be concave-convex. Grid world problem such as Mars rover applied in the paper has linear constraints instead of convex ones.

While it is true that constraints can typically be made part of the normal optimisation process in RL, by encapsulating them into the reward function, it can often be much easier to specify constraints directly, which is the setting this paper considers. The reviewers were positive about the motivation and execution of this paper, and were all in favour of accepting the paper. I would suggest already motivating this setting, at least somewhat, in the abstract, to help interesting readers find and appreciate this paper more easily.

Through theoretical analysis in TD-MPC implementation leads to persistent value and experiments, we argue that this issue is deeply rooted overestimation. It is also empirically observed that the performance in the structural policy mismatch between the data generation of TD-MPC2 is far from satisfactory at some policy that is always bootstrapped by the planner and high-dimensional locomotion tasks [33]. This phenomenon the learned policy prior. To mitigate such a mismatch in is closely connected to, yet distinct from, the well-known a minimalist way, we propose a policy regularization term overestimation bias arising from function approximation reducing out-of-distribution (OOD) queries, thereby improving errors and error accumulation in temporal difference learning value learning. Our method involves minimum changes [39, 37, 7]. More precisely, we identify the underlying on top of existing frameworks and requires no additional issue as policy mismatch. The behavior policy generated by computation. Extensive experiments demonstrate that the the MPC planner governs data collection, creating a buffered proposed approach improves performance over baselines data distribution that does not directly align with the learned such as TD-MPC2 by large margins, particularly in 61-DoF value or policy prior.