Recently, various studies have leveraged Large Language Models (LLMs) to help decision-making and planning in environments, and try to align the LLMs' knowledge with the world conditions. Nonetheless, the capacity of LLMs to continuously acquire environmental knowledge and adapt in an open world remains uncertain. In this paper, we propose an approach to spur LLMs to explore the open world, gather experiences, and learn to improve their task-solving capabilities. In this approach, a multi-round feedback-revision mechanism is utilized to encourage LLMs to actively select appropriate revision actions guided by feedback information from the environment. This facilitates exploration and enhances the model's performance. Besides, we integrate sub-task relabeling to assist LLMs in maintaining consistency in sub-task planning and help the model learn the combinatorial nature between tasks, enabling it to complete a wider range of tasks through training based on the acquired exploration experiences. By evaluation in Minecraft, an open-ended sandbox world, we demonstrate that our approach LLaMA-Rider enhances the efficiency of the LLM in exploring the environment, and effectively improves the LLM's ability to accomplish more tasks through finetuning with merely 1.3k instances of collected data, showing minimal training costs compared to the baseline using reinforcement learning. Recently, significant advancements and successes have been achieved in the performance of Large Language Models (LLMs) in attaining human-like intelligence (OpenAI, 2023). Given the powerful capability of LLMs, many research works have started utilizing their abilities to assist intelligent agents in decision-making in the environments (Yao et al., 2023; Huang et al., 2022a; Li et al., 2022; Singh et al., 2023), and have found that LLMs possess a certain level of abilities for planning and accomplishing various tasks (Wang et al., 2023b). However, the knowledge that LLMs rely on comes from the language corpus used during pre-training, and there may be discrepancies between this knowledge and specific environments (Ahn et al., 2022). To ground LLMs to environments, some studies design Figure 1. Spurring LLaMA to explore specific mechanisms through prompt engineering to provide the open world. However, LLMs do not improve or acquire new knowledge in environments.