Efficient Algorithms for Learning to Control Bandits with Unobserved Contexts

Contextual bandits are commonly used for sequential decision-making with finitely many control actions. In this setting, available context observations can be utilized in a tractable way, thanks to the linearity of the relationship between the reward and the context vectors. The arms provide rewards depending on the contexts that represent their individual characteristics. The range of real-world applications is notably extensive, including personalized recommendations for Mobile Context-Aware Recommender Systems and mobile-health interventions [1, 2, 3]. To get satisfactory performances in bandits, the exploration-exploitation trade-off must be addressed. The theoretical analysis of efficient policies for the multi-armed bandits goes back to algorithms that decide based on Upper-Confident-Bounds (UCB) [4]. In fact, UCB employs an optimistic approximate of the unknown reward based on the history of observations, to allow an appropriate degree of exploration. Further theoretical results for UCB in contextual bandits, as well as in other settings, are available in the literature [5, 6, 7, 8, 9]. Posterior sampling is another ubiquitous reinforcement learning algorithm that effectively balances exploitation versus exploration.