This work presents a novel tactile perception-based method, named T-NT, for performing the needle-threading task, an application of deformable linear object (DLO) manipulation. This task is divided into two main stages: Tail-end Finding and Tail-end Insertion. In the first stage, the agent traces the contour of the thread twice using vision-based tactile sensors mounted on the gripper fingers. The two-run tracing is to locate the tail-end of the thread. In the second stage, it employs a tactile-guided reinforcement learning (RL) model to drive the robot to insert the thread into the target needle eyelet. The RL model is trained in a Unity-based simulated environment. The simulation environment supports tactile rendering which can produce realistic tactile images and thread modeling. During insertion, the position of the poke point and the center of the eyelet are obtained through a pre-trained segmentation model, Grounded-SAM, which predicts the masks for both the needle eye and thread imprints. These positions are then fed into the reinforcement learning model, aiding in a smoother transition to real-world applications. Extensive experiments on real robots are conducted to demonstrate the efficacy of our method. More experiments and videos can be found in the supplementary materials and on the website: https://sites.google.com/view/tac-needlethreading.

Nintendo has sold a lot of Switches in the last year thanks to the console's unique ability to play games on a TV and on the go, but also thanks to The Legend of Zelda: Breath of the Wild and Super Mario Odyssey. Though they came from 30 year-old franchises, both games helped millions fall in love with them all over again. In 2018, Nintendo is setting its sights in a direction it hasn't aimed at before: the do-it yourself crowd. Nintendo Labo are a series of experiences for Switch that let you (or your kids) build cardboard objects and play games with them. Robots, fishing poles, pianos... there's a lot to build and try here.