In-Context Learning Enables Robot Action Prediction in LLMs

Robot Action You are a Franka Panda robot with a parallel gripper. We provide you with some demos in the format of Test Sample: observation>[action_1, action_2,...]. Then you will receive a new observation and you need to output a sequence of actions that match the trends in the demos. Do not output anything else. We introduce a novel framework that enables an off-the-shelf text-only LLM to directly predict robot actions through in-context learning (ICL) examples without any additional training. Our method first identifies keyframes where critical robot actions occur. We next estimate initial object poses and extract robot actions from keyframes, and both are converted into textual descriptions. Using this textual information along with the given instruction, we construct a structured prompt as ICL demonstrations, enabling the LLM to predict robot actions directly for an unseen test sample. Abstract-- Recently, Large Language Models (LLMs) have time. Through extensive experiments and analysis, RoboPrompt achieved remarkable success using in-context learning (ICL) in shows stronger performance over zero-shot and ICL baselines in the language domain.