A More Backgrounds

A.1 Distributional RL Distributional RL [2, 3, 8] is an area of RL that considers the distribution of the cumulative return Z In this paper, we estimate the quantiles of the cumulative sum cost using the quantile loss, and use them to solve the constrained optimization problem (QuantCP). A.3 The Considered Constrained Problems In this subsection, we list the problems for constrained RL. The first constrained problem is a common problem used in many previous constrained RL papers. Note that the CVaR and the quantile are two different measures for undesirable events, and the choice between the two depends on what we desire. For example, an insurance company prefers the CVaR of undesirable events to determine an insurance premium.