LangPert: Detecting and Handling Task-level Perturbations for Robust Object Rearrangement

--T ask execution for object rearrangement could be challenged by T ask-Level Perturbations (TLP)--unexpected object additions, removals, and displacements--that can disrupt underlying visual policies and fundamentally compromise task feasibility and progress. T o address these challenges, we present LangPert, a language-based framework designed to detect and mitigate TLP situations in tabletop rearrangement tasks. Lang-Pert integrates a Visual Language Model (VLM) to comprehensively monitor policy's skill execution and environmental TLP, while leveraging the Hierarchical Chain-of-Thought (HCoT) reasoning mechanism to enhance the Large Language Model (LLM)'s contextual understanding and generate adaptive, corrective skill-execution plans. Our experimental results demonstrate that LangPert handles diverse TLP situations more effectively than baseline methods, achieving higher task completion rates, improved execution efficiency, and potential generalization to unseen scenarios. Large Language Models (LLMs) have demonstrated significant potential as task planners [1], [13], [14], effectively converting a user's task description into robot skill instructions. When combined with visual perception capabilities [12] and trained on robot-specific datasets [3], [19], LLMs can serve as both task planners and environment perceivers. This enables robots to detect execution failures and re-plan skills in real-time [9], [18], [21], making LLMs a promising approach to enhance robotic autonomy in unstructured environments [20]. However, most approaches focus on task completion in static environments, overlooking the dynamic nature of real-world scenarios where unexpected workspace changes could occur due to factors such as human intervention [6]. These unpredictable disturbances [11] can significantly impact task progress and ultimately compromise task completability.