Efficient Sequential Decision Making with Large Language Models

Sequential decision making addresses the problem of adapting an agent to an unknown environment, where the agent learns through a feedback loop by repeatedly receiving contexts, selecting actions, and observing feedback. This approach has been widely applied in real-world scenarios, including recommendation systems (Li et al., 2010; Agarwal et al., 2016), healthcare Tewari and Murphy (2017); Svensson (2023), and dialogue systems (Li et al., 2016). With the significant success of large language models (LLMs) in natural language processing (Brown et al., 2020; Ouyang et al., 2022; Achiam et al., 2023), an important next step is to extend this success to sequential decision making and enhance applications therein. Existing efforts to leverage LLMs for sequential decision making focus on two directions: (i) viewing decision making as sequence modeling and re-training or finetuning large models to adapt them to unknown environments (Chen et al., 2021; Zheng et al., 2022; Reid et al., 2022; Sun et al., 2023; Raparthy et al., 2023; Lee et al., 2024), and (ii) utilizing prompt engineering and in-context learning to adapt pretrained large models to sequential decision making problems (Krishnamurthy et al., 2024). While the first approach usually achieves promising empirical results, it is hindered by the substantial computational burden associated with re-training or finetuning large models, which often contain hundreds of billions of parameters. The second approach (Krishnamurthy et al., 2024), on the other hand, has demonstrated that most in-context learning and prompt engineering methods fail to effectively adapt LLMs to sequential decision making environments, except when employing the most advanced models, i.e., GPT-4 (Achiam et al., 2023), with sophisticated prompt designs. In this paper, we propose a new approach to efficiently incorporate large pretrained models into sequential decision making environments, without the need for expensive model re-training or finetuning. We run experiments (see Figure 1 and its caption for settings) on the AmazonCat-13K dataset (Bhatia et al., 2016) and observe that: Vanilla LLMs as decision making agents exhibit strong initial performance thanks to their significant commonsense knowledge and remarkable reasoning ability.