Weaknesses: - Can we interpret the results as follows: If the TAR assumption is satisfied with positive limits, and we use MLE, then temporal interference does not cause bias. If this interpretation is correct, then it would be illuminating if the authors provide the intuitive connection between the TAR assumption and temporal interference. It is not clear if the estimations that the authors have required are feasible if the state space is large. The next natural question is how robust the results are if we use other methods for estimation. This could have been shown by providing some simulations, which is a part missing from the manuscript.

The paper studied the online experimental design problem where there are temporal dependencies between the two control policies/treatments. The novelty of the problem setup and the theoretical analysis in the paper are appreciated by all the reviewers. Although the analysis is the main contribution, the paper would be much stronger if there are meaningful experiments on toy problems to showcase the performance the online MLE-based approach vs the standard experimental design approaches.

This paper contributes a new technique for the estimation of structure in continuous time Bayesian networks, and completes the picture with an accompanying inference method and an illustration on a real-world problem. There is agreement among reviewers that this is a high quality contribution, if one takes the confidence-weighted scores from reviewers into account. As a point for improvement for the paper, we could reiterate a comment that was raised in the reviewer discussion: "[the paper] is missing reasonable and helpful experimental comparisons that are not hard to do, given that the code exists already in CTBN-RLE" and the authors are encouraged to consider broadening their experimental comparisons for a final published version.

Variational autoencoders (VAEs) are a popular framework for modeling complex data distributions; they can be efficiently trained via variational inference by maximizing the evidence lower bound (ELBO), at the expense of a gap to the exact (log-)marginal likelihood. While VAEs are commonly used for representation learning, it is unclear why ELBO maximization would yield useful representations, since unregularized maximum likelihood estimation cannot invert the data-generating process. Yet, VAEs often succeed at this task. We seek to elucidate this apparent paradox by studying nonlinear VAEs in the limit of near-deterministic decoders. We first prove that, in this regime, the optimal encoder approximately inverts the decoder---a commonly used but unproven conjecture---which we refer to as self-consistency.

Summary and Contributions: This paper studies the exploration problem in episodic reinforcement learning with kernel and neural network function approximations. The proposed algorithm is an optimistic version of least-squares value iteration, where the solution to the standard LSVI is further added by a bonus function for exploration. Under assumptions on the underlying RKHS or NTK function classes, the proposed algorithms are shown to achieve a H 2 \sqrt{T} \delta_F regret, where \delta_F depends on the effective dimension of the RKHS or NTK. First, state clearly in the introduction (maybe also abstract) that this paper makes the assumption that the transition model is characterized by the RKHS class -- I think you already did but it doesn't hurt to emphasize it. Also, revise the sentence "propose the first provable efficient RL algorithm [...] without any additional assumptions on the sampling model" (lines 44-46), e.g., by changing the term "sampling model" to be "generative model" or "simulator", as such a term is ambiguous.

This paper studies the exploration problem in episodic reinforcement learning with kernel and neural network function approximations. The authors propose a novel algorithm which is an optimistic version of least-squares value iteration, where the solution to the standard LSVI is further added by a bonus function for exploration. They derive regret bounds for this algorithm for two different function classes: RKHS and NTK. Overall, the technical contribution in this paper seems solid. Some reviewers had some concerns about the assumptions made for the analysis, especially regarding the one assuming that the Bellman optimality update lies in the RKHS.

Weaknesses: The main weakness of this paper is that a very similar problem was considered in reference [31], which has a nearly identical title. The main difference between this paper and [31] seems to be in the method: this paper uses gradient tracking, while [31] does not. Nevertheless, both papers show convergence to a neighborhood of the optimal solution, so the theoretical innovation in this paper is not sufficient for NeurIPS publication. The authors mention that the empirical results of this paper are better, but this paper seems focused on theory with a relatively brief simulation section, so it should be evaluated as a theory paper. Additionally, the results here fall short of what one wants to achieve in this setting, which is convergence to the optimal solution.

This paper is theoretical work that provides finite time analysis for decentralised TD learning. The reviewers and myself, although not anonymously, think this contribution may be significant and interesting to the community due to recent interest in the finite time analysis of TD algorithms and (linear) function approximation. We request the authors to address the changes required in the manuscript. The authors propose a distributed method for safety. The reviewers and myself were not convinced that this paper proposes a novel method, specifically, due to lack of proper comparison to previous work.

Advanced sensors and IoT devices have improved the monitoring and control of complex industrial enterprises. They have also created an interdependent fabric of geographically distributed process operations (clients) across these enterprises. Granger causality is an effective approach to detect and quantify interdependencies by examining how one client's state affects others over time. Understanding these interdependencies captures how localized events, such as faults and disruptions, can propagate throughout the system, possibly causing widespread operational impacts. However, the large volume and complexity of industrial data pose challenges in modeling these interdependencies. This paper develops a federated approach to learning Granger causality. We utilize a linear state space system framework that leverages low-dimensional state estimates to analyze interdependencies. This addresses bandwidth limitations and the computational burden commonly associated with centralized data processing. We propose augmenting the client models with the Granger causality information learned by the server through a Machine Learning (ML) function. We examine the co-dependence between the augmented client and server models and reformulate the framework as a standalone ML algorithm providing conditions for its sublinear and linear convergence rates. We also study the convergence of the framework to a centralized oracle model. Moreover, we include a differential privacy analysis to ensure data security while preserving causal insights. Using synthetic data, we conduct comprehensive experiments to demonstrate the robustness of our approach to perturbations in causality, the scalability to the size of communication, number of clients, and the dimensions of raw data. We also evaluate the performance on two real-world industrial control system datasets by reporting the volume of data saved by decentralization.

Effective traffic signal control (TSC) is crucial in mitigating urban congestion and reducing emissions. Recently, reinforcement learning (RL) has been the research trend for TSC. However, existing RL algorithms face several real-world challenges that hinder their practical deployment in TSC: (1) Sensor accuracy deteriorates with increased sensor detection range, and data transmission is prone to noise, potentially resulting in unsafe TSC decisions. (2) During the training of online RL, interactions with the environment could be unstable, potentially leading to inappropriate traffic signal phase (TSP) selection and traffic congestion. (3) Most current TSC algorithms focus only on TSP decisions, overlooking the critical aspect of phase duration, affecting safety and efficiency. To overcome these challenges, we propose a robust two-stage fuzzy approach called FuzzyLight, which integrates compressed sensing and RL for TSC deployment. FuzzyLight offers several key contributions: (1) It employs fuzzy logic and compressed sensing to address sensor noise and enhances the efficiency of TSP decisions. (2) It maintains stable performance during training and combines fuzzy logic with RL to generate precise phases. (3) It works in real cities across 22 intersections and demonstrates superior performance in both real-world and simulated environments. Experimental results indicate that FuzzyLight enhances traffic efficiency by 48% compared to expert-designed timings in the real world. Furthermore, it achieves state-of-the-art (SOTA) performance in simulated environments using six real-world datasets with transmission noise. The code and deployment video are available at the URL1