Visual servoing, the method of controlling robot motion through feedback from visual sensors, has seen significant advancements with the integration of optical flow-based methods. However, its application remains limited by inherent challenges, such as the necessity for a target image at test time, the requirement of substantial overlap between initial and target images, and the reliance on feedback from a single camera. This paper introduces Imagine2Servo, an innovative approach leveraging diffusion-based image editing techniques to enhance visual servoing algorithms by generating intermediate goal images. This methodology allows for the extension of visual servoing applications beyond traditional constraints, enabling tasks like long-range navigation and manipulation without predefined goal images. We propose a pipeline that synthesizes subgoal images grounded in the task at hand, facilitating servoing in scenarios with minimal initial and target image overlap and integrating multi-camera feedback for comprehensive task execution. Our contributions demonstrate a novel application of image generation to robotic control, significantly broadening the capabilities of visual servoing systems. Real-world experiments validate the effectiveness and versatility of the Imagine2Servo framework in accomplishing a variety of tasks, marking a notable advancement in the field of visual servoing.

In standard policy learning, a single neural-network based policy is tasked with learning both of these skills (and learning to switch between them), without any access to structures that explicitly encode the multi-modal nature of task space.Ideally, policies would be able to emergently learn to decompose tasks at different levels of abstraction, and factor the task learning into unique skills. One common approach is to try and jointly learn a set of subskills, as well as a selection function which selects a specific subskill to execute at the current time step [5]. This poses a fundamental bootstrapping issue: as the skills change and improve, the selection function must change and improve as well, which can lead to unstable training. An important observation of many optimal policies for manipulation tasks is that skills tend to be executed in sequence, without backtracking. Therefore, time itself can serve as a useful indicator for skill selection. For instance, while executing a stacking task, it is reasonable to assume that the robot will undertake the'reach' skill at the start of the task, and subsequently perform the'stack' skill towards the end of the task. Our intuition here is that selecting the'skill' according to which time-step we are currently at can be used as a good strategy for selecting