I thank the author's for their response. I would like to reemphasize that I like this paper a lot, and applaud the authors for their work. I strongly suggest an oral accept for this paper. Additionally, I've raised my score 1 point. However, I still felt that Section 3.1 was somewhat contrived, and stand by my initial criticism of this section.

The authors propose remapping value functions into a logarithmic space, leading to "logarithmic Q-learning" which is demonstrated to perform quite well in practice. This paper has by far the strongest overall scores (9, 9, 8) in my paper batch. All three reviewers are enthusiastic about the paper and its contributions and results. I am recommending that NeurIPS accept the paper for Oral presentation.

This paper introduces a learning-based visual planner for agile drone flight in cluttered environments. The proposed planner generates collision-free waypoints in milliseconds, enabling drones to perform agile maneuvers in complex environments without building separate perception, mapping, and planning modules. Learning-based methods, such as behavior cloning (BC) and reinforcement learning (RL), demonstrate promising performance in visual navigation but still face inherent limitations. BC is susceptible to compounding errors due to limited expert imitation, while RL struggles with reward function design and sample inefficiency. To address these limitations, this paper proposes an inverse reinforcement learning (IRL)-based framework for high-speed visual navigation. By leveraging IRL, it is possible to reduce the number of interactions with simulation environments and improve capability to deal with high-dimensional spaces while preserving the robustness of RL policies. A motion primitive-based path planning algorithm collects an expert dataset with privileged map data from diverse environments, ensuring comprehensive scenario coverage. By leveraging both the acquired expert and learner dataset gathered from the agent's interactions with the simulation environments, a robust reward function and policy are learned across diverse states. While the proposed method is trained in a simulation environment only, it can be directly applied to real-world scenarios without additional training or tuning. The performance of the proposed method is validated in both simulation and real-world environments, including forests and various structures. The trained policy achieves an average speed of 7 m/s and a maximum speed of 8.8 m/s in real flight experiments. To the best of our knowledge, this is the first work to successfully apply an IRL framework for high-speed visual navigation of drones.

The fields of MPC and RL consider two successful control techniques for Markov decision processes. Both approaches are derived from similar fundamental principles, and both are widely used in practical applications, including robotics, process control, energy systems, and autonomous driving. Despite their similarities, MPC and RL follow distinct paradigms that emerged from diverse communities and different requirements. Various technical discrepancies, particularly the role of an environment model as part of the algorithm, lead to methodologies with nearly complementary advantages. Due to their orthogonal benefits, research interest in combination methods has recently increased significantly, leading to a large and growing set of complex ideas leveraging MPC and RL. This work illuminates the differences, similarities, and fundamentals that allow for different combination algorithms and categorizes existing work accordingly. Particularly, we focus on the versatile actor-critic RL approach as a basis for our categorization and examine how the online optimization approach of MPC can be used to improve the overall closed-loop performance of a policy.

The main works [10, 11] attack the same realistic fraud detection Adversarial attacks pose a significant threat to data-driven engine called BankSealer [9]. In both works, the authors systems, and researchers have spent considerable resources rightfully consider domain-specific challenges generally absent studying them. Despite its economic relevance, this trend in other adversarial works, such as the intricate feature largely overlooked the issue of credit card fraud detection. To engineering process performed in fraud detection. However, address this gap, we propose a new threat model that demonstrates they operate under the assumption that fraudsters can access the limitations of existing attacks and highlights the the customers' transaction history. As the authors point out, necessity to investigate new approaches. We then design a this may be achieved through the introduction of malware into new adversarial attack for credit card fraud detection, employing the victim's devices. However, this considerably increases the reinforcement learning to bypass classifiers. This attack, difficulty of performing any attack, as fraudsters must first called FRAUD-RLA, is designed to maximize the attacker's compromise the customer's device and observe past transaction reward by optimizing the exploration-exploitation tradeoff history, which constitutes a significantly more complex and working with significantly less required knowledge than undertaking than stealing or cloning a card.

Maximum entropy reinforcement learning (MaxEnt-RL) has become the standard approach to RL due to its beneficial exploration properties. Traditionally, policies are parameterized using Gaussian distributions, which significantly limits their representational capacity. Diffusion-based policies offer a more expressive alternative, yet integrating them into MaxEnt-RL poses challenges--primarily due to the intractability of computing their marginal entropy. To overcome this, we propose Diffusion-Based Maximum Entropy RL (DIME). DIME leverages recent advances in approximate inference with diffusion models to derive a lower bound on the maximum entropy objective. Additionally, we propose a policy iteration scheme that provably converges to the optimal diffusion policy. Our method enables the use of expressive diffusion-based policies while retaining the principled exploration benefits of MaxEnt-RL, significantly outperforming other diffusion-based methods on challenging high-dimensional control benchmarks. It is also competitive with state-of-the-art non-diffusion based RL methods while requiring fewer algorithmic design choices and smaller update-to-data ratios, reducing computational complexity.

The alteration of the climate in various areas of the world is of increasing concern since climate stability is a necessary condition for human survival as well as every living organism. The main reason of climate change is the greenhouse effect caused by the accumulation of carbon dioxide in the atmosphere. In this paper, we design a networked system underpinned by compartmental dynamical thermodynamics to circulate the atmospheric carbon dioxide. Specifically, in the carbon dioxide emitter compartment, we develop an initial-condition-dependent finite-time stabilizing controller that guarantees stability within a desired time leveraging the system property of affinity in the control. Then, to compensate for carbon emissions we show that a cultivation of microalgae with a volume 625 times bigger than the one of the carbon emitter is required. To increase the carbon uptake of the microalgae, we implement the nonaffine-in-the-control microalgae dynamical equations as an environment of a state-of-the-art library for reinforcement learning (RL), namely, Stable-Baselines3, and then, through the library, we test the performance of eight RL algorithms for training a controller that maximizes the microalgae absorption of carbon through the light intensity. All the eight controllers increased the carbon absorption of the cultivation during a training of 200,000 time steps with a maximum episode length of 200 time steps and with no termination conditions. This work is a first step towards approaching net zero as a classical and learning-based network control problem. The source code is publicly available.

This paper addresses the challenge of anti-jamming in moving reactive jamming scenarios. The moving reactive jammer initiates high-power tracking jamming upon detecting any transmission activity, and when unable to detect a signal, resorts to indiscriminate jamming. This presents dual imperatives: maintaining hiding to avoid the jammer's detection and simultaneously evading indiscriminate jamming. Spread spectrum techniques effectively reduce transmitting power to elude detection but fall short in countering indiscriminate jamming. Conversely, changing communication frequencies can help evade indiscriminate jamming but makes the transmission vulnerable to tracking jamming without spread spectrum techniques to remain hidden. Current methodologies struggle with the complexity of simultaneously optimizing these two requirements due to the expansive joint action spaces and the dynamics of moving reactive jammers. To address these challenges, we propose a parallelized deep reinforcement learning (DRL) strategy. The approach includes a parallelized network architecture designed to decompose the action space. A parallel exploration-exploitation selection mechanism replaces the $\varepsilon $-greedy mechanism, accelerating convergence. Simulations demonstrate a nearly 90\% increase in normalized throughput.

This paper introduces a novel agent-based approach for enhancing existing portfolio strategies using Proximal Policy Optimization (PPO). Rather than focusing solely on traditional portfolio construction, our approach aims to improve an already high-performing strategy through dynamic rebalancing driven by PPO and Oracle agents. Our target is to enhance the traditional 60/40 benchmark (60% stocks, 40% bonds) by employing the Regret-based Sharpe reward function. To address the impact of transaction fee frictions and prevent signal loss, we develop a transaction cost scheduler. We introduce a future-looking reward function and employ synthetic data training through a circular block bootstrap method to facilitate the learning of generalizable allocation strategies. We focus on two key evaluation measures: return and maximum drawdown. Given the high stochasticity of financial markets, we train 20 independent agents each period and evaluate their average performance against the benchmark. Our method not only enhances the performance of the existing portfolio strategy through strategic rebalancing but also demonstrates strong results compared to other baselines.

Robot learning requires a considerable amount of high-quality data to realize the promise of generalization. However, large data sets are costly to collect in the real world. Physics simulators can cheaply generate vast data sets with broad coverage over states, actions, and environments. However, physics engines are fundamentally misspecified approximations to reality. This makes direct zero-shot transfer from simulation to reality challenging, especially in tasks where precise and force-sensitive manipulation is necessary. Thus, fine-tuning these policies with small real-world data sets is an appealing pathway for scaling robot learning. However, current reinforcement learning fine-tuning frameworks leverage general, unstructured exploration strategies which are too inefficient to make real-world adaptation practical. This paper introduces the Simulation-Guided Fine-tuning (SGFT) framework, which demonstrates how to extract structural priors from physics simulators to substantially accelerate real-world adaptation. Specifically, our approach uses a value function learned in simulation to guide real-world exploration. We demonstrate this approach across five real-world dexterous manipulation tasks where zero-shot sim-to-real transfer fails. We further demonstrate our framework substantially outperforms baseline fine-tuning methods, requiring up to an order of magnitude fewer real-world samples and succeeding at difficult tasks where prior approaches fail entirely. Last but not least, we provide theoretical justification for this new paradigm which underpins how SGFT can rapidly learn high-performance policies in the face of large sim-to-real dynamics gaps. Project webpage: https://weirdlabuw.github.io/sgft/{weirdlabuw.github.io/sgft}