Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximation

Reinforcement learning (RL) [43] has emerged as a powerful framework for sequential decision-making in dynamic and uncertain environments. While traditional RL methods, predominantly focused on maximizing the expected return, have seen significant advancements through approaches such as Q-learning [37, 25] and policy gradients [28, 10], they often fall short in real-world scenarios demanding strict risk control, such as financial investment [9], medical treatment [16], and automous driving [11]. The significance of comprehending risk management in RL has led to the emergence of Risk-Sensitive RL (RSRL). Unlike risk-neutral RL, which primarily focuses on maximizing expected returns, RSRL seeks to optimize risk metrics, such as entropy risk measures (ERM) [17, 18] or conditional value-at-risk (CVaR) [46], of the possible cumulative reward which emphasizes its distributional characteristics. However, traditional RL framework based on Q-learning which typically considers the mean of reward-to-go and corresponding Bellman equation, cannot efficiently capture the characteristics of the cumulative reward's distribution. Therefore, there has been an upsurge of interest in Distributional RL (DisRL) due to its capacity to understand the intrinsic distributional attributes of cumulative rewards, which has already achieved significant empirical success in risk-sensitive tasks [8, 14, 30, 45, 34].