In economics, Return on Investment (ROI) is a financial metric used to evaluate the profitability of an investment relative to its cost. The concept of ROI originates from the work of [5] and is widelyregardedasavaluable metricbythemajority ofmarketingmanagers [4].

Large language models (LLMs) have been increasingly used to interact with external environments (e.g., games, compilers, APIs) as goal-driven agents.

Inthis section, we provide additional discussions of applying decision-focused learning toMDPs problems. Specifically, the assumption on smooth policy is similar to the idea of soft Q-learning [12] and soft actor-critic [13]proposed by Haarnoja et al. The randomly initiated neural network uses ReLU layers asnonlinearity followed byalinear layer intheend. Training parameters Across all three examples, we consider the discounted setting where the discount factor isγ = 0.95. Torelax the optimal policygivenbythe RL solver,we relax the Bellman equation used to run value-iteration by relaxing all the argmax and max operators in theBellman equation tosoftmax with temperature0.1,i.e., weuseSOFTMAX(0.1 Q-values)to replace all the argmax over Q values.

To resolve the first challenge, we propose to sample anestimate ofthefirst-order andsecond-order derivativestoapproximate theoptimality andKKT conditions.