Could humanoid robots be heading for the battlefield? I've come to an industrial space in a tech-heavy area of San Francisco expecting to see a menacing humanoid robot solider doing something combat-like: the future of land-based warfare, perhaps. Instead, the black shiny faceless Phantom robot is engaged in free play, manipulating a bunch of coloured kids blocks. We need data from it just interacting with its environment [and] this is today's menu, explains Sankaet Pathak, co-founder and CEO of two-year-old start-up Foundation Robotics, which is developing Phantom for military and civilian applications. Later he pushes its 80kg steel-covered body around the room to demonstrate its stability and shows me how it walks.

Large Vision Models trained on internet-scale data have demonstrated strong capabilities in segmenting and semantically understanding object parts, even in cluttered, crowded scenes. However, while these models can direct a robot toward the general region of an object, they lack the geometric understanding required to precisely control dexterous robotic hands for 3D grasping. To overcome this, our key insight is to leverage simulation with a force-closure grasping generation pipeline that understands local geometries of the hand and object in the scene. Because this pipeline is slow and requires ground-truth observations, the resulting data is distilled into a diffusion model that operates in real-time on camera point clouds. By combining the global semantic understanding of internet-scale models with the geometric precision of a simulation-based locally-aware force-closure, \our achieves high-performance semantic grasping without any manually collected training data. For visualizations of this please visit our website at https://ifgrasping.github.io/

A single AI model trained to control numerous robotic bodies can operate unfamiliar hardware and adapt eerily well to serious injuries. A four-legged robot that keeps crawling even after all four of its legs have been hacked off with a chainsaw is the stuff of nightmares for most people. For Deepak Pathak, cofounder and CEO of the startup Skild AI, the dystopian feat of adaptation is an encouraging sign of a new, more general kind of robotic intelligence. "This is something we call an omni-bodied brain," Pathak tells me. His startup developed the generalist artificial intelligence algorithm to address a key challenge with advancing robotics: "Any robot, any task, one brain.

In this work, we propose a parameter continuation method for the optimization of neural networks. There is a close connection between parameter continuation, homotopies, and curriculum learning. The methods we propose here are theoretically justified and practically effective for several problems in deep neural networks. In particular, we demonstrate better generalization performance than state-of-the-art optimization techniques such as ADAM for supervised and unsupervised learning tasks.

To train generalist robot policies, machine learning methods often require a substantial amount of expert human teleoperation data. An ideal robot for humans collecting data is one that closely mimics them: bimanual arms and dexterous hands. However, creating such a bimanual teleoperation system with over 50 DoF is a significant challenge. To address this, we introduce Bidex, an extremely dexterous, low-cost, low-latency and portable bimanual dexterous teleoperation system which relies on motion capture gloves and teacher arms. We compare Bidex to a Vision Pro teleoperation system and a SteamVR system and find Bidex to produce better quality data for more complex tasks at a faster rate. Additionally, we show Bidex operating a mobile bimanual robot for in the wild tasks. The robot hands (5k USD) and teleoperation system (7k USD) is readily reproducible and can be used on many robot arms including two xArms (16k USD). Website at https://bidex-teleop.github.io/

Combining the mobility of legged robots with the manipulation skills of arms has the potential to significantly expand the operational range and enhance the capabilities of robotic systems in performing various mobile manipulation tasks. Existing approaches are confined to imprecise six degrees of freedom (DoF) manipulation and possess a limited arm workspace. In this paper, we propose a novel framework, RoboDuet, which employs two collaborative policies to realize locomotion and manipulation simultaneously, achieving whole-body control through interactions between each other. Surprisingly, going beyond the large-range pose tracking, we find that the two-policy framework may enable cross-embodiment deployment such as using different quadrupedal robots or other arms. Our experiments demonstrate that the policies trained through RoboDuet can accomplish stable gaits, agile 6D end-effector pose tracking, and zero-shot exchange of legged robots, and can be deployed in the real world to perform various mobile manipulation tasks. Our project page with demo videos is at https://locomanip-duet.github.io .

Neural networks (NN) can be divided into two broad categories, recurrent and non-recurrent. Both types of neural networks are popular and extensively studied, but they are often treated as distinct families of machine learning algorithms. In this position paper, we argue that there is a closer relationship between these two types of neural networks than is normally appreciated. We show that many common neural network models, such as Recurrent Neural Networks (RNN), Multi-Layer Perceptrons (MLP), and even deep multi-layer transformers, can all be represented as iterative maps. The close relationship between RNNs and other types of NNs should not be surprising. In particular, RNNs are known to be Turing complete, and therefore capable of representing any computable function (such as any other types of NNs), but herein we argue that the relationship runs deeper and is more practical than this. For example, RNNs are often thought to be more difficult to train than other types of NNs, with RNNs being plagued by issues such as vanishing or exploding gradients. However, as we demonstrate in this paper, MLPs, RNNs, and many other NNs lie on a continuum, and this perspective leads to several insights that illuminate both theoretical and practical aspects of NNs.

Dexterous manipulation has been a long-standing challenge in robotics. While machine learning techniques have shown some promise, results have largely been currently limited to simulation. This can be mostly attributed to the lack of suitable hardware. In this paper, we present LEAP Hand, a low-cost dexterous and anthropomorphic hand for machine learning research. In contrast to previous hands, LEAP Hand has a novel kinematic structure that allows maximal dexterity regardless of finger pose. LEAP Hand is low-cost and can be assembled in 4 hours at a cost of 2000 USD from readily available parts. It is capable of consistently exerting large torques over long durations of time. We show that LEAP Hand can be used to perform several manipulation tasks in the real world -- from visual teleoperation to learning from passive video data and sim2real. LEAP Hand significantly outperforms its closest competitor Allegro Hand in all our experiments while being 1/8th of the cost. We release detailed assembly instructions, the Sim2Real pipeline and a development platform with useful APIs on our website at https://leap-hand.github.io/

Researchers at Carnegie Mellon University's School of Computer Science and the University of California have designed a system that enables a small, low-cost robot to climb and descend stairs, traverse uneven and varied terrain, walk across gaps, and even operate in the dark. The research could be a step toward solving existing challenges facing legged robots and bringing them into people's homes, say researchers. A paper supporting the research - Legged Locomotion in Challenging Terrains Using Egocentric Vision - will be presented at the upcoming Conference on Robot Learning in Auckland, New Zealand. "Empowering small robots to climb stairs and handle a variety of environments is crucial to developing robots that will be useful in people's homes as well as search-and-rescue operations," says Deepak Pathak, an Assistant Professor at Carnegie Mellon's Robotics Institute. "This system creates a robust and adaptable robot that could perform many everyday tasks."

Researchers at Carnegie Mellon University's School of Computer Science and the University of California, Berkeley, have designed a robotic system that enables a low-cost and relatively small legged robot to climb and descend stairs nearly its height; traverse rocky, slippery, uneven, steep and varied terrain; walk across gaps; scale rocks and curbs; and even operate in the dark. "Empowering small robots to climb stairs and handle a variety of environments is crucial to developing robots that will be useful in people's homes as well as search-and-rescue operations," said Deepak Pathak, an assistant professor in the Robotics Institute. "This system creates a robust and adaptable robot that could perform many everyday tasks." The team put the robot through its paces, testing it on uneven stairs and hillsides at public parks, challenging it to walk across stepping stones and over slippery surfaces, and asking it to climb stairs that for its height would be akin to a human leaping over a hurdle. The researchers trained the robot with 4,000 clones of it in a simulator, where they practiced walking and climbing on challenging terrain.