Abstract--Industrial Internet of Things (IIoT) applications demand efficient task offloading to handle heavy data loads with minimal latency. Mobile Edge Computing (MEC) brings computation closer to devices to reduce latency and server load, optimal performance requires advanced optimization techniques. We propose a novel solution combining Adaptive Particle Swarm Optimization (APSO) with Reinforcement Learning, specifically Soft Actor Critic (SAC), to enhance task offloading decisions in MEC environments. This hybrid approach leverages swarm intelligence and predictive models to adapt to dynamic variables such as human interactions and environmental changes. Our method improves resource management and service quality, achieving optimal task offloading and resource distribution in IIoT edge computing.

Edge sensing and computing is rapidly becoming part of intelligent infrastructure architecture leading to operational reliance on such systems in disaster or emergency situations. In such scenarios there is a high chance of power supply failure due to power grid issues, and communication system issues due to base stations losing power or being damaged by the elements, e.g., flooding, wildfires etc. Mobile edge computing in the form of unmanned aerial vehicles (UAVs) has been proposed to provide computation offloading from these devices to conserve their battery, while the use of UAVs as relay network nodes has also been investigated previously. This paper considers the use of UAVs with further constraints on power and connectivity to prolong the life of the network while also ensuring that the data is received from the edge nodes in a timely manner. Reinforcement learning is used to investigate numerous scenarios of various levels of power and communication failure. This approach is able to identify the device most likely to fail in a given scenario, thus providing priority guidance for maintenance personnel. The evacuations of a rural town and urban downtown area are also simulated to demonstrate the effectiveness of the approach at extending the life of the most critical edge devices.

This paper presents an approach that combines Human-In-The-Loop Reinforcement Learning (HITL RL) with principles derived from music theory to facilitate real-time generation of musical compositions. HITL RL, previously employed in diverse applications such as modelling humanoid robot mechanics and enhancing language models, harnesses human feedback to refine the training process. In this study, we develop a HILT RL framework that can leverage the constraints and principles in music theory. In particular, we propose an episodic tabular Q-learning algorithm with an epsilon-greedy exploration policy. The system generates musical tracks (compositions), continuously enhancing its quality through iterative human-in-the-loop feedback. The reward function for this process is the subjective musical taste of the user.

Sample efficiency and systematic generalization are two long-standing challenges in reinforcement learning. Previous studies have shown that involving natural language along with other observation modalities can improve generalization and sample efficiency due to its compositional and open-ended nature. However, to transfer these properties of language to the decision-making process, it is necessary to establish a proper language grounding mechanism. One approach to this problem is applying inductive biases to extract fine-grained and informative representations from the observations, which makes them more connectable to the language units. We provide architecture-level inductive biases for modularity and sparsity mainly based on Neural Production Systems (NPS). Alongside NPS, we assign a central role to memory in our architecture. It can be seen as a high-level information aggregator which feeds policy/value heads with comprehensive information and simultaneously guides selective attention in NPS through attentional feedback. Our results in the BabyAI environment suggest that the proposed model's systematic generalization and sample efficiency are improved significantly compared to previous models. An extensive ablation study on variants of the proposed method is conducted, and the effectiveness of each employed technique on generalization, sample efficiency, and training stability is specified.

Retrieval-augmented generation (RAG) is extensively utilized to incorporate external, current knowledge into large language models, thereby minimizing hallucinations. A standard RAG pipeline may comprise several components, such as query rewriting, document retrieval, document filtering, and answer generation. However, these components are typically optimized separately through supervised fine-tuning, which can lead to misalignments between the objectives of individual modules and the overarching aim of generating accurate answers in question-answering (QA) tasks. Although recent efforts have explored reinforcement learning (RL) to optimize specific RAG components, these approaches often focus on overly simplistic pipelines with only two components or do not adequately address the complex interdependencies and collaborative interactions among the modules. To overcome these challenges, we propose treating the RAG pipeline as a multi-agent cooperative task, with each component regarded as an RL agent. Specifically, we present MMOA-RAG, a Multi-Module joint Optimization Algorithm for RAG, which employs multi-agent reinforcement learning to harmonize all agents' goals towards a unified reward, such as the F1 score of the final answer. Experiments conducted on various QA datasets demonstrate that MMOA-RAG improves the overall pipeline performance and outperforms existing baselines. Furthermore, comprehensive ablation studies validate the contributions of individual components and the adaptability of MMOA-RAG across different RAG components and datasets. The code of MMOA-RAG is on https://github.com/chenyiqun/MMOA-RAG.

The factorization of state-action value functions for Multi-Agent Reinforcement Learning (MARL) is important. Existing studies are limited by their representation capability, sample efficiency, and approximation error. To address these challenges, we propose, ResQ, a MARL value function factorization method, which can find the optimal joint policy for any state-action value function through residual functions. ResQ masks some state-action value pairs from a joint state-action value function, which is transformed as the sum of a main function and a residual function. ResQ can be used with mean-value and stochastic-value RL.

Originality: To the best of my knowledge the model of general user retention dynamics and corresponding statements evidencing negative feedback loops are novel contributions to the literature in sequential fairness works. The contributions of the paper would be clearer if citations were provided for methods and models introduced in earlier works (for example, I suggest adding citations for the fairness criteria in lines 149-158, for user departure models in 197-208, and for the statement in lines 173-174, if applicable). Since the full related work is deferred to the appendix, I see no need to cite [2, 3, 7, 10, 15, 16] without distinction between them. More context on what these works do and how they relate to your work is useful for readers to contextualize your contributions; please expand on the discussion of these papers. Quality: The simple and unifying model of sequential decision making presented is very valuable in my opinion.

There was some disagreement among the reviewers. In a subsequent discussion, the positive points outweighed the negative ones and I am happy to support acceptance. The topic is important, the approach is reasonable and the empirical contribution, despite some caveats, is convincing. That being said, I ask the authors to incorporate the reviewers' suggestions in the final version of their paper.

Summary and Contributions: This paper considers the fundamental problem of computing a fixed point of a stochastic contractive operator. Formally, this paper supposes that there is an operator H from R d to R d such that H(x) – x_* _c \leq \gamma x – x_* _c for some norm \cdot _c, some gamma \in (0, 1), and fixed _x_* and that given any point x can compute H(x) w where w is mean zero noise, the magnitude of which depends on x. The paper then considers the natural stochastic approximation (SA) algorithm x_k 1 x_k epsilon_k (H(x_k) – x_k w_k) for some sequences step sizes epsilon_k and stochastic mean-0 w_k, the norm of which depends on x_k. The paper provides general convergence bounds for this algorithm in terms of the norm in which H contracts and the bound on the noise vector w_k in terms of x_k. Formally, the paper supposes that conditioned up to iteration k, E w_k 2_n \leq A (1 x_k _n 2) for some norm and provides bounds on the convergence of the method.

Weaknesses: The main weakness of the work is in its presentation. For a reader that is not intimately familiar with the background material, Section 2 is not self contained, and the significance of the concept of mixed NE vs pure NE is not explained. But perhaps the main area of the paper that would benefit greatly from additional discussion is the experiments section, which currently features a very large Figure 4 (consider cutting down to half the current size) and little discussion of the results themselves. When does the proposed method work better vs worse than the baselines, and is there intuition for why? Some videos of the learned policies would also nicely supplement the results.