There are lots of ways to incorporate AI into robots, starting with improving how they are trained to do tasks. But using large language models to give instructions, as Google has done, is particularly interesting. The robotics startup Figure went viral a year ago for a video in which humans gave instructions to a humanoid on how to put dishes away. Around the same time, a startup spun off from OpenAI, called Covariant, built something similar for robotic arms in warehouses. I saw a demo where you could give the robot instructions via images, text, or video to do things like "move the tennis balls from this bin to that one."

In 2012, Amazon quietly acquired a robotics startup called Kiva Systems, a move that dramatically improved the efficiency of its ecommerce operations and kickstarted a wider revolution in warehouse automation. Last week, the ecommerce giant announced another deal that could prove similarly profound, agreeing to hire the founders of Covariant, a startup that has been testing ways for AI to automate more of the picking and handling of a wide range of physical objects. Covariant may have found it challenging to commercialize AI-infused industrial robots given the high costs and sharp competition involved; the deal, which will also see Amazon license Covariant's models and data, could bring about another revolution in ecommerce--one that might prove hard for any competitor to match given Amazon's vast operational scale and data trove. The deal is also an example of a Big Tech company acquiring core talent and expertise from an AI startup without actually buying the company outright. Amazon came to a similar agreement with the startup Adept in June.

The news: In the summer of 2021, OpenAI quietly shuttered its mulrobotics team, announcing that progress was being stifled by a lack of data necessary to train robots in how to move and reason using artificial intelligence. Now three of OpenAI's early research scientists say the startup they spun off in 2017, called Covariant, has solved that problem. They've unveiled a system that combines the reasoning skills of large language models with the physical dexterity of an advanced robot. How it works: The new model, called RFM-1, was trained on years of data collected from Covariant's small fleet of item-picking robots, as well as words and videos from the internet. Users can prompt the model using five different types of input: text, images, video, robot instructions, and measurements.

The new model, called RFM-1, was trained on years of data collected from Covariant's small fleet of item-picking robots that customers like Crate & Barrel and Bonprix use in warehouses around the world, as well as words and videos from the internet. In the coming months, the model will be released to Covariant customers. The company hopes the system will become more capable and efficient as it's deployed in the real world. In a demonstration I attended last week, Covariant cofounders Peter Chen and Pieter Abbeel showed me how users can prompt the model using five different types of input: text, images, video, robot instructions, and measurements. For example, show it an image of a bin filled with sports equipment, and tell it to pick up the pack of tennis balls. The robot can then grab the item, generate an image of what the bin will look like after the tennis balls are gone, or create a video showing a bird's-eye view of how the robot will look doing the task.

Peter Chen, CEO of the robot software company Covariant, sits in front of a chatbot interface resembling the one used to communicate with ChatGPT. "Show me the tote in front of you," he types. In reply, a video feed appears, revealing a robot arm over a bin containing various items--a pair of socks, a tube of chips, and an apple among them. The chatbot can discuss the items it sees--but also manipulate them. When WIRED suggests Chen ask it to grab a piece of fruit, the arm reaches down, gently grasps the apple, and then moves it to another bin nearby.

The human brain is wired to be able to learn new things--and in all kinds of different ways, from imitating others to watching online explainer videos. What if robots could do the same thing? It is a question that ACM Prize recipient Pieter Abbeel, professor at the University of California, Berkeley and director of the Berkeley Robot Learning Lab, has spent his career researching. Here, we speak with Abbeel about his work and about the techniques he has developed to make it easier to teach robots. Let's start with deep reinforcement learning and the method you developed called Trust Region Policy Optimization.

Covariant, an AI robotics company, has announced it has raised an additional $75 million in Series C funds, bringing its total funding to $222 million. Returning investors Radical Ventures and Index Ventures co-led the round, which also saw additional funding from returning investors Canada Pension Plan Investment Board and Amplify Partners. The round also welcomed new investors Gates Frontier Holdings, AIX Ventures, and Northgate Capital. The funding will be used to ensure today's leading retailers and their logistics providers are able to deploy robotic picking quickly and without disruption to their current operations, Covariant stated. This comes at a time when retail executives are eager to invest in AI-powered robotic automation: according to a Covariant-led research survey from February 2023, more than 80% of retail leaders see automation as a key solution for navigating operational uncertainty in an unpredictable marketplace – and 98% plan to further invest in AI Robotics in 2023 despite current economic conditions.

OpenAI, the company behind ChatGPT, has become the fastest-growing consumer application in history. With more than 30 executives, engineers, and researchers leaving the company to start their own companies, OpenAI has raised over US$1 billion in financing and created the "OpenAI Mafia", a powerful network of talent, social connections, and capital opportunities. This new generation of AI companies is driving a new round of technological frenzy and investment opportunities, and OpenAI is dedicated to helping humans realize their beautiful vision with an elite team. The OpenAI Mafia is the new generation of AI companies founded by OpenAI employees in the past five years, and is set to revolutionize the AI industry and shape the future of AI technology. Anthropic is an AI company founded in 2021 by Dario and Daniela Amodei, former vice presidents of OpenAI.

Find open roles in Artificial Intelligence (AI), Machine Learning (ML), Natural Language Processing (NLP), Computer Vision (CV), Data Engineering, Data Analytics, Big Data, and Data Science in general, filtered by job title or popular skill, toolset and products used.

Here we take a standard method found in physics used to remove known background functions from data, for example removing the spectra associated with amorphous radiators from those associated with lattice-structure radiators to obtain enhanced spectra, that is without noise (Timm 1969). Here we consider the variations associated with covariants as similar background data to be removed, so as to observe if there is any variation associated with EDV. Similar to partial correlations, removing the background signal of a potential covariant allows us to visually evaluate the specific impact a variable of interest has on the target variable. This involves fitting the control data and the target (e.g., the size of training data and LAS) and then dividing the target variable by the predicted values from this fit. This normalized data is then used to fit a second potential covariant which too is used to divide the normalized target variable values. This can be repeated for any number of covariants. Ultimately a normalized version of the target variable is left and the control target of interest (e.g., EDV) is evaluated against these values and if a trend is still observed, it is evidence that this variable has an impact on the target variable even with the variance associated with these covariants removed. This technique ultimately acts as a way of tempering correlations we calculate and gives us a means of disentangling contributions that might not be caught by partial correlation calculations.