Eye-in-Finger: Smart Fingers for Delicate Assembly and Disassembly of LEGO

-- Manipulation and insertion of small and tight-toleranced objects in robotic assembly remain a critical challenge for vision-based robotics systems due to the required precision and cluttered environment. Conventional global or wrist-mounted cameras often suffer from occlusions when either assembling or disassembling from an existing structure. T o address the challenge, this paper introduces "Eye-in-Finger", a novel tool design approach that enhances robotic manipulation by embedding low-cost, high-resolution perception directly at the tool tip. We validate our approach using LEGO assembly and disassembly tasks, which require the robot to manipulate in a cluttered environment and achieve sub-millimeter accuracy and robust error correction due to the tight tolerances. Experimental results demonstrate that our proposed system enables real-time, fine corrections to alignment error, increasing the tolerance of calibration error from 0.4mm to up to 2.0mm for the LEGO manipulation robot. Humans rely on vision for overall spatial perception but depend on tactile sensing for fine-grained, high-precision interactions [1]. For example, when threading a needle, placing a microchip on a circuit board, or performing delicate sutures in surgery, visual guidance provides an initial estimate, while tactile feedback refines positioning and detailed operations.