Multi-Task Reinforcement Learning with Mixture of Orthogonal Experts

Hessian Center for Artificial Intelligence (Hessian.ai), Multi-Task Reinforcement Learning (MTRL) tackles the long-standing problem of endowing agents with skills that generalize across a variety of problems. To this end, sharing representations plays a fundamental role in capturing both unique and common characteristics of the tasks. Tasks may exhibit similarities in terms of skills, objects, or physical properties while leveraging their representations eases the achievement of a universal policy. Nevertheless, the pursuit of learning a shared set of diverse representations is still an open challenge. In this paper, we introduce a novel approach for representation learning in MTRL that encapsulates common structures among the tasks using orthogonal representations to promote diversity. Our method, named Mixture Of Orthogonal Experts (MOORE), leverages a Gram-Schmidt process to shape a shared subspace of representations generated by a mixture of experts. When task-specific information is provided, MOORE generates relevant representations from this shared subspace. We assess the effectiveness of our approach on two MTRL benchmarks, namely MiniGrid and MetaWorld, showing that MOORE surpasses related baselines and establishes a new state-of-the-art result on MetaWorld. Despite the success of RL, generalizing the learned policy to a broader set of related tasks remains an open challenge. Multi-Task Reinforcement Learning (MTRL) is introduced to scale up the RL framework, holding the promise of enabling learning a universal policy capable of addressing multiple tasks concurrently. To this end, sharing knowledge is key in MTRL (Teh et al., 2017; D'Eramo et al., 2020; Sodhani et al., 2021; Sun et al., 2022). However, deciding upon the kind of knowledge to share, and the set of tasks to share that knowledge, is crucial for designing an efficient MTRL algorithm.