CV aR minimization algorithm to account for the covariate shift in the data-generating distribution. The advantage of risk-sensitive (either risk-seeking or risk-averse) objectives in machine learning, however, is not limited to tasks involving social considerations. Indeed, there exists a rich body of works which implicitly propose to minimize risk-sensitive measures of loss, as a technique to better optimize the standard expected loss.

We deeply appreciate constructive and insightful comments from the reviewers. We will clarify this point in the revised version. On the other hand, we agree to the reviewer's point We believe that R1 refers to (Line 42), where we point to [30]. By contrast, we focus on nonsequential scenarios where such concerns do not arise. Apparent lack of tailored analysis for risk-sensitivity.

We consider the problem of estimating the Optimized Certainty Equivalent (OCE) risk from independent and identically distributed (i.i.d.) samples. For the classic sample average approximation (SAA) of OCE, we derive mean-squared error as well as concentration bounds (assuming sub-Gaussianity). Further, we analyze an efficient stochastic approximation-based OCE estimator, and derive finite sample bounds for the same. To show the applicability of our bounds, we consider a risk-aware bandit problem, with OCE as the risk. For this problem, we derive bound on the probability of mis-identification. Finally, we conduct numerical experiments to validate the theoretical findings.

Virtually all machine learning tasks are characterized using some form of loss function, and "good performance" is typically stated in terms of a sufficiently small average loss, taken over the random draw of test data. While optimizing for performance on average is intuitive, convenient to analyze in theory, and easy to implement in practice, such a choice brings about trade-offs. In this work, we survey and introduce a wide variety of non-traditional criteria used to design and evaluate machine learning algorithms, place the classical paradigm within the proper historical context, and propose a view of learning problems which emphasizes the question of "what makes for a desirable loss distribution?" in place of tacit use of the expected loss.

The systematic minimization of the quantifiable uncertainty, or risk [22], is one of the core objectives in all disciplines involving decision-making, e.g., economics and finance. Within machine learning contexts, strategies for risk-aversion have been most actively studied under sequential decision-making and reinforcement learning frameworks [21, 8], giving birth to a number of algorithms based on Markov decision processes (MDPs) and multi-armed bandits. In those works, various risk-averse measures of loss have been used as a minimization objective, instead of the risk-neutral expected loss; popular risk measures include entropic risk [21, 6, 7], mean-variance [39, 13, 28], and a slightly more modern alternative known as conditional value-at-risk (CVaR [15, 10, 42]). Yet, with growing interest to the societal impacts of machine intelligence, the importance of risk-aversion under non-sequential scenarios has also been spotlighted recently. For instance, Williamson and Menon [45] give an axiomatic characterization of the fairness risk measures, and propose a convex fairness-aware objective based on CVaR.