We introduce a novel policy learning method that integrates analytical gradients from differentiable environments with the Proximal Policy Optimization (PPO) algorithm.

Specifically,at each iteration the black-box function is evaluated at one or more input locations that maximizes anacquisition function onthemodel.

In economics, Return on Investment (ROI) is a financial metric used to evaluate the profitability of an investment relative to its cost. The concept of ROI originates from the work of [5] and is widelyregardedasavaluable metricbythemajority ofmarketingmanagers [4].

Real-world problems such as ad allocation and matching havebeen extensively studied under the lens of combinatorial optimization. In several applications, uncertainty in the input appears naturally and this has led to the study of online stochastic optimization models for such problems.