This paper studies constrained optimal control, where the goal is to produce a policy that maximizes an objective function subject to a constraint. The authors provide great motivation for this setting, explaining why the constraint cannot simply be included as a large negative reward. They detail challenges in solving this problem, especially if the initial policy does not satisfy the constraint. They also note a clever extension of their method, where they use the constraint to define the objective, by setting the constraint to indicate whether the task is solved. Their algorithm builds upon CEM: at each iteration, if there are no feasible policies, they maximize the constraint function for the policies with the largest objective; otherwise, they maximize the objective function for feasible policies.

In many sequential decision-making problems, the decision-maker has access to contextual information that can guide personalized actions. Contextual Multi-Armed Bandit (CMAB) algorithms formalize this setting, extending traditional multi-armed bandits by incorporating side information (e.g., user attributes, environmental factors, or historical states) to inform arm selection [Lattimore and Szepesv ari, 2020]. This contextual adaptability makes CMABs especially valuable in dynamic, high-stakes applications like personalized healthcare, targeted advertising, and adaptive pricing. A growing body of literature explores leveraging observational (offline) data (collected from past interactions without active experimentation) to initialize and accelerate learning in CMABs [Ten-nenholtz et al., 2021, Hao et al., 2023]. Offline data provides a valuable source of information on context-arm-outcome relationships, which can substantially reduce the need for costly, risky, or even unethical online exploration in high-stakes settings. More importantly, observational data can play a critical role in identifying a sparse set ofinformative features that are most predictive of treatment responses. In fact, in high-dimensional settings, not all features equally influence the effectiveness of an action. Many features may be irrelevant or weakly correlated with the outcome, introducing noise and exacerbating the sample efficiency of online learning algorithms.

We study offline constrained reinforcement learning (RL) with general function approximation. We aim to learn a policy from a pre-collected dataset that maximizes the expected discounted cumulative reward for a primary reward signal while ensuring that expected discounted returns for multiple auxiliary reward signals are above predefined thresholds. Existing algorithms either require fully exploratory data, are computationally inefficient, or depend on an additional auxiliary function classes to obtain an $ε$-optimal policy with sample complexity $O(ε^{-2})$. In this paper, we propose an oracle-efficient primal-dual algorithm based on a linear programming (LP) formulation, achieving $O(ε^{-2})$ sample complexity under partial data coverage. By introducing a realizability assumption, our approach ensures that all saddle points of the Lagrangian are optimal, removing the need for regularization that complicated prior analyses. Through Lagrangian decomposition, our method extracts policies without requiring knowledge of the data-generating distribution, enhancing practical applicability.

The success of Deepseek-R1 has drawn the LLM community's attention to reinforcement learning (RL) methods like GRPO. However, such rule-based 0/1 outcome reward methods lack the capability to regulate the intermediate reasoning processes during chain-of-thought (CoT) generation, leading to severe overthinking phenomena. In response, recent studies have designed reward functions to reinforce models' behaviors in producing shorter yet correct completions. Nevertheless, we observe that these length-penalty reward functions exacerbate RL training instability: as the completion length decreases, model accuracy abruptly collapses, often occurring early in training. To address this issue, we propose a simple yet effective solution GRPO-$λ$, an efficient and stabilized variant of GRPO, which dynamically adjusts the reward strategy by monitoring the correctness ratio among completions within each query-sampled group. A low correctness ratio indicates the need to avoid length penalty that compromises CoT quality, triggering a switch to length-agnostic 0/1 rewards that prioritize reasoning capability. A high ratio maintains length penalties to boost efficiency. Experimental results show that our approach avoids training instability caused by length penalty while maintaining the optimal accuracy-efficiency trade-off. On the GSM8K, GPQA, MATH-500, AMC 2023, and AIME 2024 benchmarks, it improves average accuracy by 1.48% while reducing CoT sequence length by 47.3%.

The original AIXI reinforcement learning agent, intended as a near ly parameter-free formal gold standard for artificial general intelligence (AGI), is a Cartesian dualist that believes it is interacting with an environment from the outside, in the sense that its policy is fixed and not overwritten by anything that happens in the environment, though its actions can certainly adapt based on the percepts it receives. This is frequently compared to a person playin g a video game, who certainly does not believe he is being simulated by the game b ut rather interacts with it only by observing the screen and pressing b uttons. In contrast, it would presumably be important for an AGI to be aware t hat it exists within its environment (the universe) and its computations ar e therefore subject to the laws of physics. With this in mind, we investigate versio ns of the AIXI agent [Hut00] that treat the action sequence a on a similar footing to the percept sequence e, meaning that the actions are considered as explainable by the same rules generating the percepts. The most obvious idea is to use the universal distribution to model the joint (action/percept) dis tribution (even though actions are selected by the agent). Although this is the mos t direct way to transform AIXI into an embedded agent, it does not appear to h ave been analyzed in detail; in particular, it is usually assumed (but not proven) to fail (often implicitly, without distinguishing the universal sequence and environment distributions, e.g.

Large-language-model (LLM) agents excel at reactive dialogue but struggle with proactive, goal-driven interactions due to myopic decoding and costly planning. We introduce DialogXpert, which leverages a frozen LLM to propose a small, high-quality set of candidate actions per turn and employs a compact Q-network over fixed BERT embeddings trained via temporal-difference learning to select optimal moves within this reduced space. By tracking the user's emotions, DialogXpert tailors each decision to advance the task while nurturing a genuine, empathetic connection. Across negotiation, emotional support, and tutoring benchmarks, DialogXpert drives conversations to under $3$ turns with success rates exceeding 94\% and, with a larger LLM prior, pushes success above 97\% while markedly improving negotiation outcomes. This framework delivers real-time, strategic, and emotionally intelligent dialogue planning at scale. Code available at https://github.com/declare-lab/dialogxpert/

PPO - BR establishes a new paradigm in adaptive RL by fusing exploration and convergence signals into a single bounded trust region -- a theoretically - grounded innovation (Theorem 1) that outperforms 5 SOTA baselines with <2% overhead (Fig 3). This work bridges a critical gap in phase - aware learning, enabling real - world deployment in safety - critical systems like robotic surgery (Appendix E) within a single theoretically - grounded trust region mechanism (Theorem 1), achieving 29.1% faster convergence: (1) Entropy - driven expansion (ϵ) promotes exploration in high - uncertainty states, while (2) reward - guided contraction (ϵ) enforces stability during convergence (Theorem 1). On 6 diverse benchmarks (MuJoCo/Atari/sparse - reward), PPO - BR achieves: 29.1% fa ster convergence (p < 0.001, Wilcoxon test), 2.3 lower reward variance vs PPO (Fig 3), and <1.8% runtime overhead with just 5 lines of code change (Algorithm 1). PPO - BR's plug - and - play simplicity and theoretical guarantees (Lemma 2) make it ready - to - deplo y in safety - critical systems -- from surgical robotics to autonomous drones -- where adaptive stability is non - negotiable . In contrast to recent methods such as Group Relative Policy Optimization (GRPO), PPO - BR offers a unified entropy - reward adaptive mechanism applicable to both language models and general reinforcement learning environments.

We present a hierarchical policy-learning framework that enables a legged humanoid to cooperatively carry extended loads with a human partner using only haptic cues for intent inference. At the upper tier, a lightweight behavior-cloning network consumes six-axis force/torque streams from dual wrist-mounted sensors and outputs whole-body planar velocity commands that capture the leader's applied forces. At the lower tier, a deep-reinforcement-learning policy, trained under randomized payloads (0-3 kg) and friction conditions in Isaac Gym and validated in MuJoCo and on a real Unitree G1, maps these high-level twists to stable, under-load joint trajectories. By decoupling intent interpretation (force -> velocity) from legged locomotion (velocity -> joints), our method combines intuitive responsiveness to human inputs with robust, load-adaptive walking. We collect training data without motion-capture or markers, only synchronized RGB video and F/T readings, employing SAM2 and WHAM to extract 3D human pose and velocity. In real-world trials, our humanoid achieves cooperative carry-and-move performance (completion time, trajectory deviation, velocity synchrony, and follower-force) on par with a blindfolded human-follower baseline. This work is the first to demonstrate learned haptic guidance fused with full-body legged control for fluid human-humanoid co-manipulation. Code and videos are available on the H2-COMPACT website.

Specifically, it is a policy gradient method, often used for deep learning when the policy network is very large. The predecessor to PPO, Trust Region Policy Optimization (TRPO), was published in 2015 by Schulman et al . It addressed the instability issue of another algorithm, the Deep Q - Network (DQN).

Modern industry-scale data centers need to manage a large number of virtual machines (VMs). Due to the continual creation and release of VMs, many small resource fragments are scattered across physical machines (PMs). To handle these fragments, data centers periodically reschedule some VMs to alternative PMs, a practice commonly referred to as VM rescheduling. Despite the increasing importance of VM rescheduling as data centers grow in size, the problem remains understudied. We first show that, unlike most combinatorial optimization tasks, the inference time of VM rescheduling algorithms significantly influences their performance, due to dynamic VM state changes during this period. This causes existing methods to scale poorly. Therefore, we develop a reinforcement learning system for VM rescheduling, VM2RL, which incorporates a set of customized techniques, such as a two-stage framework that accommodates diverse constraints and workload conditions, a feature extraction module that captures relational information specific to rescheduling, as well as a risk-seeking evaluation enabling users to optimize the trade-off between latency and accuracy. We conduct extensive experiments with data from an industry-scale data center. Our results show that VM2RL can achieve a performance comparable to the optimal solution but with a running time of seconds. Code and datasets are open-sourced: https://github.com/zhykoties/VMR2L_eurosys, https://drive.google.com/drive/folders/1PfRo1cVwuhH30XhsE2Np3xqJn2GpX5qy.