We introduce Multistage Conditional Compositional Optimization (MCCO) as a new paradigm for decision-making under uncertainty that combines aspects of multistage stochastic programming and conditional stochastic optimization. MCCO minimizes a nest of conditional expectations and nonlinear cost functions. It has numerous applications and arises, for example, in optimal stopping, linear-quadratic regulator problems, distributionally robust contextual bandits, as well as in problems involving dynamic risk measures. The naïve nested sampling approach for MCCO suffers from the curse of dimensionality familiar from scenario tree-based multistage stochastic programming, that is, its scenario complexity grows exponentially with the number of nests. We develop new multilevel Monte Carlo techniques for MCCO whose scenario complexity grows only polynomially with the desired accuracy.

In order to perform a wide search over the space of the trees, it is critical to solve this projection problem fast.

While therewardcompensation mechanism isunknown,the learner can adapt his (her) decision to the past reward feedback so as to maximize the sum of rewards.

This is meaningful because in most applications, the time complexities for evaluating gradients at different points are of the same magnitude.

PageRank (PR), an algorithm originally proposed by Page et al. for ranking web-pages [1] has found manysuccessful applications, including community detection [2,3],linkprediction [4]and recommendersystemdesign[5,6].

Namely,thattheempirical lossofall the functions in the class is -close to the true loss. Finally, we develop a set of tools for calculating the approximate description length of classes of functions that can be presented as a composition of linear function classes and non-linear functions.

B.1 Learningdetails All of our code and experiments are implemented in Python using the PyTorch [5]deep learning framework.

Our main result is a matching upper and lower bound for the secretary ranking problem.

We need some more notation in order to linearize the value function.

Many potential applications of reinforcement learning (RL) have intractably-large state spaces.