Navigating the Social Welfare Frontier: Portfolios for Multi-objective Reinforcement Learning

In this paper, we study a reinforcement learning (RL) setting where a deployed policy impacts multiple stakeholders in different ways. Each stakeholder is associated with a unique reward function, and the goal is to train a policy that adequately aggregates their preferences. This setting, which is often modeled using multi-objective reinforcement learning (MORL), arises in many RL applications, such as fair resource allocation in healthcare [32], cloud computing [27, 18] and communication networks [36, 7]. Recently, with the rise of large language models (LLMs), reinforcement learning from human feedback (RLHF) techniques that reflect the preferences of heterogeneous individuals have also been explored [6, 38, 26]. Preference aggregation in such scenarios is often achieved by choosing a social welfare function, which takes the utilities of multiple stakeholders as input and outputs a scalar value representing the overall welfare [37, 9, 32, 13, 38, 26, 6]. However, selecting the appropriate social welfare function is a nontrivial task, as each function embodies a different notion of social welfare and can lead to vastly different outcomes for the involved stakeholders. In this work, we focus on a class of social welfare functions known as generalized p-means, a widely used class of social welfare functions in algorithmic fairness and social choice theory.