Stochastic Approximation for Risk-aware Markov Decision Processes

The analysis of complex systems such as inventory control, financial markets, waste-to-energy plants and computer networks is difficult because of the inherent uncertainties in these systems. Risk-aware optimization offers a possible remedy by giving stronger reliability guarantees than the risk-neutral case. Furthermore, it allows expression of the risk attitude of the decision maker. Risk awareness is especially important in sequential decision making because of the dynamic nature of the uncertainty. Markov decision processes (MDPs) introduced by Bellman in [10] provide a mathematical framework for modeling sequential decision making in situations where outcomes are partly random and partly under the control the decision maker. However, in many cases the exact model of the underlying Markov decision process is not known and one can only observe the trajectory of states, actions, and rewards/costs.