This survey presents an overview of methods for learning from video (LfV) in the context of reinforcement learning (RL) and robotics. We focus on methods capable of scaling to large internet video datasets and, in the process, extracting foundational knowledge about the world's dynamics and physical human behaviour. Such methods hold great promise for developing general-purpose robots. We open with an overview of fundamental concepts relevant to the LfV-for-robotics setting. This includes a discussion of the exciting benefits LfV methods can offer (e.g., improved generalization beyond the available robot data) and commentary on key LfV challenges (e.g., missing information in video and LfV distribution shifts). Our literature review begins with an analysis of video foundation model techniques that can extract knowledge from large, heterogeneous video datasets. Next, we review methods that specifically leverage video data for robot learning. Here, we categorise work according to which RL knowledge modality (KM) benefits from the use of video data. We additionally highlight techniques for mitigating LfV challenges, including reviewing action representations that address missing action labels in video. Finally, we examine LfV datasets and benchmarks, before concluding with a discussion of challenges and opportunities in LfV. Here, we advocate for scalable foundation model approaches that can leverage the full range of internet video data, and that target the learning of the most promising RL KMs: the policy and dynamics model. Overall, we hope this survey will serve as a comprehensive reference for the emerging field of LfV, catalysing further research in the area and facilitating progress towards the development of general-purpose robots.

People take pictures and videos of the Boston Dynamics robot Spot during an event in Lisbon in 2019. People take pictures and videos of the Boston Dynamics robot Spot during an event in Lisbon in 2019. Boston Dynamics and five other robotics companies have signed an open letter saying what many of us were already nervously hoping for anyway: Let's not weaponize general-purpose robots. The six leading tech firms -- including Agility Robotics, ANYbotics, Clearpath Robotics, Open Robotics and Unitree -- say advanced robots could result in huge benefits in our work and home lives but that they may also be used for nefarious purposes. "Untrustworthy people could use them to invade civil rights or to threaten, harm, or intimidate others," the companies said.

"I am a visionary," says an AI, kicking off the latest installment of NVIDIA's I AM AI video series. Launched in 2017, I AM AI has become the iconic opening for GTC keynote addresses by NVIDIA founder and CEO Jensen Huang. Each video, with its AI-created narration and soundtrack, documents the newest advances in artificial intelligence and their impact on the world. The latest, which debuted at GTC last week, showcases how NVIDIA technologies enable AI to take on complex tasks in the world's most challenging environments, from farms and traffic intersections to museums and research labs. Here's a sampling of the groundbreaking AI innovations featured in the video.

But it falls short of the definition of artificial general intelligence (AGI), which would be a machine capable of understanding the world as well as any human. In the 1950s, researchers including AI pioneer Herbert A. Simon were convinced that AGI would exist within the next few decades. Since then, AGI has proven to be a daunting, perhaps even impossible-to-achieve milestone. Writing in The Guardian, roboticist Alan Winfield claimed the gulf between modern computing and AGI is as wide as the gulf between current space flight and faster-than-light travel. Still, others insist that AGI is drawing close within reach.

Vancouver's Sanctuary AI, a company focused on creating the world's first human-like intelligence in general-purpose robots, today announced the successful closing of an oversubscribed $75.5 million Series A funding. Investors in the massive financing round include Bell, Evok Innovations, Export Development Canada, Magna, SE Health, Verizon Ventures, and Workday Ventures. Using breakthrough technology in artificial intelligence (AI), cognition, and robotics, Sanctuary will improve the quality of the work experience, assist humans with difficult or dangerous tasks, create new jobs, bring new opportunities to those who might be less capable of physical work, and reduce the impact of labour shortages around the world. The strategic industry investors reflect applications for human-like intelligence in general-purpose robots across a wide range of industry verticals and tasks. Founded in 2018 by Geordie Rose, Suzanne Gildert, Olivia Norton, and Ajay Agrawal, Sanctuary is on a mission to create the world's first human-like intelligence in general-purpose robots that will help us work more safely, efficiently, and sustainably.

Today, the house-cleaning Roomba seems almost ubiquitous, but in a recent essay, its inventor, Joe Jones, recalls his wrong prediction in the 1980s that "in three to five years, robots will be everywhere doing all sorts of jobs." For decades, he notes, "robots never managed to find their way out of the laboratory." Jones makes several good points about why some robotics companies fail: they fail to perform a valuable task, they fail to do the task today, or they fail to do the task for less. Robotic solutions need to be extremely simple. That's why the Roomba worked: the usefulness of an autonomous vacuum brought real robotics into many people's (and animals') homes with simple sensing, behavior-based programming, and mobility.

The concept of autonomous machines dates back to medieval times, but the research into the practical and potential use of robots began only in the 20th century. Today, there are numerous scholars, inventors, engineers, and technicians that are working to develop machines that mimic human behaviour and manage tasks in a human-like fashion. While artificial intelligence plays a crucial role in the development and advancement of robotics, the rise of general-purpose robots poses a question of whether robotics has begun to lag AI. People often confuse robotics for industrial automation and academia and research. While for most high-end research robots, it has deep learning embedded into them such as computer vision linked objects, feature detection and classification, industrial robots are beginning to include the maturity of camera-based object detection and classification.

Over a year ago, OpenAI, the San Francisco–based for-profit AI research lab, announced that it had trained a robotic hand to manipulate a cube with remarkable dexterity. That might not sound earth-shattering. But in the AI world, it was impressive for two reasons. First, the hand had taught itself how to fidget with the cube using a reinforcement-learning algorithm, a technique modeled on the way animals learn. Second, all the training had been done in simulation, but it managed to successfully translate to the real world.

It's the undisputed heavyweight champion of AI held up as proof of machines hell-bent on the destruction of humanity. In my experience seeing Atlas do parkour and backflips, and four-legged Spot robots get pushed around by humans, Boston Dynamics is a close second. These robots fascinate and terrify people. If facial recognition software and Amazon's Alexa are held up as popular examples of surveillance capitalism, Boston Dynamics videos are usually shoved in my face by people afraid of these robots' mobility and physical prowess. This is partially due to the advanced robotics and unique design, and partially due to the success of a YouTube campaign over the course of the past six months in which each video sucks up millions of views.

Our latest robotics techniques allow robot controllers, trained entirely in simulation and deployed on physical robots, to react to unplanned changes in the environment as they solve simple tasks. That is, we've used these techniques to build closed-loop systems rather than open-loop ones as before. The simulator need not match the real-world in appearance or dynamics; instead, we randomize relevant aspects of the environment, from friction to action delays to sensor noise. Our new results provide more evidence that general-purpose robots can be built by training entirely in simulation, followed by a small amount of self-calibration in the real world. This robot was trained in simulation with dynamics randomization to push a puck to a goal.