Federated Learning (FL) is a distributed machine learning strategy, developed for settings where training data is owned by distributed devices and cannot be shared. FL circumvents this constraint by carrying out model training in distribution. The parameters of these local models are shared intermittently among participants and aggregated to enhance model accuracy. This strategy has been rapidly adopted by the industry in efforts to overcome privacy and resource constraints in model training. However, the application of FL to real-world settings brings additional challenges associated with heterogeneity between participants. Research into mitigating these difficulties in FL has largely focused on only two types of heterogeneity: the unbalanced distribution of training data, and differences in client resources. Yet more types of heterogeneity are becoming relevant as the capability of FL expands to cover more complex problems, from the tuning of LLMs to enabling machine learning on edge devices. In this work, we discuss a novel type of heterogeneity that is likely to become increasingly relevant in future applications: this is preference heterogeneity, emerging when clients learn under multiple objectives, with different importance assigned to each objective on different clients. In this work, we discuss the implications of this type of heterogeneity and propose FedPref, a first algorithm designed to facilitate personalised FL in this setting. We demonstrate the effectiveness of the algorithm across different problems, preference distributions and model architectures. In addition, we introduce a new analytical point of view, based on multi-objective metrics, for evaluating the performance of FL algorithms in this setting beyond the traditional client-focused metrics. We perform a second experimental analysis based in this view, and show that FedPref outperforms compared algorithms.

Artificial intelligence (AI) is expected to significantly enhance radio resource management (RRM) in sixth-generation (6G) networks. However, the lack of explainability in complex deep learning (DL) models poses a challenge for practical implementation. This paper proposes a novel explainable AI (XAI)- based framework for feature selection and model complexity reduction in a model-agnostic manner. Applied to a multi-agent deep reinforcement learning (MADRL) setting, our approach addresses the joint sub-band assignment and power allocation problem in cellular vehicle-to-everything (V2X) communications. We propose a novel two-stage systematic explainability framework leveraging feature relevance-oriented XAI to simplify the DRL agents. While the former stage generates a state feature importance ranking of the trained models using Shapley additive explanations (SHAP)-based importance scores, the latter stage exploits these importance-based rankings to simplify the state space of the agents by removing the least important features from the model input. Simulation results demonstrate that the XAI-assisted methodology achieves 97% of the original MADRL sum-rate performance while reducing optimal state features by 28%, average training time by 11%, and trainable weight parameters by 46% in a network with eight vehicular pairs.

What if artificial intelligence could not only solve problems for which it was trained but also learn to teach itself to solve new problems (i.e., meta-learn)? In this study, we demonstrate that a pre-trained transformer fine-tuned with reinforcement learning over multiple episodes develops the ability to solve problems that it has never encountered before - an emergent ability called In-Context Reinforcement Learning (ICRL). This powerful meta-learner not only excels in solving unseen in-distribution environments with remarkable sample efficiency, but also shows strong performance in out-of-distribution environments. In addition, we show that it exhibits robustness to the quality of its training data, seamlessly stitches together behaviors from its context, and adapts to non-stationary environments. These behaviors demonstrate that an RL-trained transformer can iteratively improve upon its own solutions, making it an excellent general-purpose problem solver.

The primary motivation for the work is not well supported. Certainly, cities do manage thousands of intersections. While unquantified, it is not clear that the cost of training individually would surpass that of the degradation seen in the multi-env setting. These two statements seem to be conflicting. In section 5.1, the single-env results, it is not clear that FRAP is only applicable in 37 of the 112 cases.

There was a consensus among reviewers that the paper should be accepted. The paper provides a novel application of attention-based networks to traffic light control. The universality of the architecture allows out-of-the box application of a trained policy to new intersections without the need for adaptation. Overall, this paper seems to have the potential to have a strong impact.

Additional Feedback: medium points: table 1: the "Lower Bounds" method doesn't have "Time complexity" or "Space complexity" entries? Also why is it separated from the other prior work? Is this assuming something the others baselines in Table 1 aren't? If so, is this a fair comparison then? Everything except red (epsilonQL) seems to perform the same.

The work has clear positives: paper presents a novel algorithm that achieves low regret novel consideration of adaptive discretization for model-based RL (prior work focuses on the model-free case) important practical focus on computational resources novel theory However, there are significant issues as well: - the experiments are not were presented or discussed within the context of the rest of the paper: they appear to contradict the main messages of the paper - there is confusion over how the experiments were implemented and evaluated (e.g., proper averaging over independent runs, fair treatment of hyperparameters etc). See R1 for more details - the proposed algorithm exhibits space complexity that monotonically increases; the authors suggested to just cap it - poor discussion of model-based RL with function approximation (linear dyna, recent deep learning approaches etc) - related no clear argument why we would explore adaptive discretization approaches compared to other approaches. The paper is doing something different than the majority of the community---that can be good but it should be directly addressed Summary of the discussion. The reviewers thought the experiments considerably weakened the paper, and it would be best if they were removed from the paper. The strongest advocate of the paper had low confidence and not much to say much during discussion.

Strengths: The deep information maximization objective combined with noise contrastive estimation (InfoNCE) is a fairly new unsupervised learning loss that has yet to be thoroughly explored in deep reinforcement learning. The main value of the paper is the study of the representations learned when optimizing the InfoNCE loss and how those representations can be used for continual learning. Moreover, the paper introduces a novel architecture that uses the action information as part of the InfoNCE loss. These two ideas are novel and, to my knowledge, they haven't been presented in the literature before. In terms of significance, there has been growing interest in the representations learned by the InfoNCE loss in the context of reinforcement learning; see, Oord, Li, and Vinyals (2018), Anand et.

This paper proposes a method to apply noise contrastive estimation for future state prediction as an auxiliary task for RL agents. The authors clearly explain their formulation and through toy experiments show it working as intended. There are some empirical improvements in performance in simple continual learning settings and also in Procgen. The author response contains very useful ablation studies and connection to prior work which I hope the authors consider adding to the final draft, as well as acknowledgement of moving theory sections to make exposition clearer.

The paper studies the problem of policy poisoning in batch reinforcement learning and control where the learner estimates the model of the world from batch data set, and finds an optimal policy with respect to the learned model. The attacker modifies the data by the means of modifying the reward entries to make the agent learn a target policy. The paper presents a framework for solving batch policy poison attacks on two standard victims. The theoretical and experimental results show some evidence for the feasibility of policy poisoning attacks. Overall, I think this is an interesting paper that is motivated under a realistic adversarial setting where the attacker can alter the reward (instead of altering the dynamics of the world) to change the optimal policy to an adversarial target policy. The paper is easy to read due to its clear organization.