Federated Reinforcement Learning (FedRL) improves sample efficiency Despite its promise, most FedRL frameworks [8, 10, 18, 50] operate while preserving privacy; however, most existing studies under the assumption of agent homogeneity (i.e., identical assume homogeneous agents, limiting its applicability in real-world policy networks and training configurations), which significantly scenarios. This paper investigates FedRL in black-box settings with limits FedRL's applicability in real-world scenarios. This limitation heterogeneous agents, where each agent employs distinct policy is particularly acute in resource-constrained environments, such as networks and training configurations without disclosing their internal in edge environments, where agents have limited power and need details. Knowledge Distillation (KD) is a promising method to adapt network structures and training strategies based on their for facilitating knowledge sharing among heterogeneous models, operational conditions to achieve effective training [47]. In addition, but it faces challenges related to the scarcity of public datasets and existing FedRL frameworks typically operate under a white-box limitations in knowledge representation when applied to FedRL. To paradigm, where models are openly shared among participants.