We say a real-valued random variable X is -sub-Gaussian if it its mean is zero and for all " 2 R we have E[exp("X)] exp Such assumptions on the noise variables are frequently used in bandit optimization. Typically, in kernelized bandits, we assume that unknown f 2F k(D;B)= {f 2H k(D): kfkk B}, where Hk(D) is the reproducing kernel Hilbert space of functions associated with the given positive-definite kernel function. Typically, the learner knows Fk(D;B), meaning that both k(,) and B are considered as input to the learner's algorithm. We outline some commonly used kernel functions k: D D! R, that we also consider: Linear kernel: klin(x,x0)= xTx0, Squared exponential kernel: kSE(x,x0)=exp kx x0k2 2l2, Matérn kernel: kMat(x,x0)= 2 Maximum information gain is a kernel-dependent quantity that measures the complexity of the given function class. It has first been introduced in [40], and since then it has been used in numerous works on Gaussian process bandits.

We consider the problem of optimizing a black-box function based on noisy bandit feedback. Kernelized bandit algorithms have shown strong empirical and theoretical performance for this problem. They heavily rely on the assumption that the model is well-specified, however, and can fail without it. Instead, we introduce a misspecified kernelized bandit setting where the unknown function can be -uniformly approximated by a function with a bounded norm in some Reproducing Kernel Hilbert Space (RKHS).

Pseudo-games, or abstract economies [4], are optimization problems that are closely related to min-max Stackelberg games, but which are technically not games, as noted by Facchinei and Kanzow [26, 27], because each player's strategy set is not fixed at the outset (i.e., before they have to make a decision), but instead depends on the other players' choices. In this appendix, we formally define two-player, zero-sum pseudo-games,10 and discuss how they differ from min-max Stackelberg games. We also define the equilibrium concept par excellence of pseudo-games, namely generalized Nash equilibrium, and juxtapose its definition with vanilla Nash equilibrium. A two-player, zero-sum pseudo-game comprises two players, with respective payoff functions f(x,y)and f(x,y), and respective strategy spaces given by the correspondences X: Y X and Y: X Y, i.e., set valued mappings that depend on the choice the other player takes. Pseudo-games are closely related to min-max Stackelberg games, as they both comprise agents with the same objectives and the same space of feasible strategy profiles, namely {(x,y) 2 X Y |8 k 2 [K],gk(x,y) 0}.

Why One-step and IQL are imitation-based methods? The core difference between RL-based and imitation-based methods is that RL-based methods learn a value function of policy π while imitation-based methods don't. Learning the value function of π requires off-policy evaluation of π (i.e., learning Qπ or Vπ), which is prone to distribution shift. The policy evaluation and policy improvement will also affect each other as they are coupled. Imitation-based methods don't learn Qπ or Vπ, but some of them do learn a value function.

We study the design of computationally efficient online learning algorithms under smoothed analysis. In this setting, at every step an adversary generates a sample from an adaptively chosen distribution whose density is upper bounded by 1/ times the uniform density. Given access to an offline optimization (ERM) oracle, we give the first computationally efficient online algorithms whose sublinear regret depends only on the pseudo/VC dimension dof the class and the smoothness parameter .