Inverse Risk-Sensitive Reinforcement Learning

HE modeling and learning of human decision-making behavior is increasingly becoming important as critical systems begin to rely more on automation and artificial intelligence. Yet, in this task we face a number of challenges, not least of which is the fact that humans are known to behave in ways that are not completely rational. There is mounting evidence to support the fact that humans often use reference points--e.g., the status quo or former experiences or recent expectaions about the future that are otherwise perceived to be related to the decision the human is making [1], [2]. It has also been observed that their decisions are impacted by their perception of the external world (exogenous factors) and their present state of mind (endogenous factors) as well as how the decision is framed or presented [3]. The success of descriptive behavioral models in capturing human behavior has long been touted by the psychology community and, more recently, by the economics community. In the engineering context, humans have largely been modeled, under rationality assumptions, from the so-called normative point of view where things are modeled as they ought to be, which is counter to a descriptive as is point of view. However, risk-sensitivity in the context of learning to control stochastic dynamical systems (see, e.g., [4], [5]) has been fairly extensively explored in engineering. Many of these approaches are targeted at mitigating risks due to uncertainties in controlling a system such as a plant or robot where risk-aversion is captured by leveraging techniques such as exponential utility functions or minimizing mean-variance criteria. Complex risk-sensitive behavior arising from human interaction with automation is only recently coming into focus.