The ability to learn from large batches of autonomously collected data for policy improvement -- a paradigm we refer to as batch online reinforcement learning -- holds the promise of enabling truly scalable robot learning by significantly reducing the need for human effort of data collection while getting benefits from self-improvement. Yet, despite the promise of this paradigm, it remains challenging to achieve due to algorithms not being able to learn effectively from the autonomous data. For example, prior works have applied imitation learning and filtered imitation learning methods to the batch online RL problem, but these algorithms often fail to efficiently improve from the autonomously collected data or converge quickly to a suboptimal point. This raises the question of what matters for effective batch online RL in robotics. Motivated by this question, we perform a systematic empirical study of three axes -- (i) algorithm class, (ii) policy extraction methods, and (iii) policy expressivity -- and analyze how these axes affect performance and scaling with the amount of autonomous data. Through our analysis, we make several observations. First, we observe that the use of Q-functions to guide batch online RL significantly improves performance over imitation-based methods. Building on this, we show that an implicit method of policy extraction -- via choosing the best action in the distribution of the policy -- is necessary over traditional policy extraction methods from offline RL. Next, we show that an expressive policy class is preferred over less expressive policy classes. Based on this analysis, we propose a general recipe for effective batch online RL. We then show a simple addition to the recipe of using temporally-correlated noise to obtain more diversity results in further performance gains. Our recipe obtains significantly better performance and scaling compared to prior methods.

A long-standing goal in robot learning is to develop methods for robots to acquire new skills autonomously. While reinforcement learning (RL) comes with the promise of enabling autonomous data collection, it remains challenging to scale in the real-world partly due to the significant effort required for environment design and instrumentation, including the need for designing reset functions or accurate success detectors. On the other hand, imitation learning (IL) methods require little to no environment design effort, but instead require significant human supervision in the form of collected demonstrations. To address these shortcomings, recent works in autonomous IL start with an initial seed dataset of human demonstrations that an autonomous policy can bootstrap from. While autonomous IL approaches come with the promise of addressing the challenges of autonomous RL as well as pure IL strategies, in this work, we posit that such techniques do not deliver on this promise and are still unable to scale up autonomous data collection in the real world. Through a series of real-world experiments, we demonstrate that these approaches, when scaled up to realistic settings, face much of the same scaling challenges as prior attempts in RL in terms of environment design. Further, we perform a rigorous study of autonomous IL methods across different data scales and 7 simulation and real-world tasks, and demonstrate that while autonomous data collection can modestly improve performance, simply collecting more human data often provides significantly more improvement. Our work suggests a negative result: that scaling up autonomous data collection for learning robot policies for real-world tasks is more challenging and impractical than what is suggested in prior work. We hope these insights about the core challenges of scaling up data collection help inform future efforts in autonomous learning.

Intelligent instruction-following robots capable of improving from autonomously collected experience have the potential to transform robot learning: instead of collecting costly teleoperated demonstration data, large-scale deployment of fleets of robots can quickly collect larger quantities of autonomous data that can collectively improve their performance. However, autonomous improvement requires solving two key problems: (i) fully automating a scalable data collection procedure that can collect diverse and semantically meaningful robot data and (ii) learning from non-optimal, autonomous data with no human annotations. To this end, we propose a novel approach that addresses these challenges, allowing instruction-following policies to improve from autonomously collected data without human supervision. Our framework leverages vision-language models to collect and evaluate semantically meaningful experiences in new environments, and then utilizes a decomposition of instruction following tasks into (semantic) language-conditioned image generation and (non-semantic) goal reaching, which makes it significantly more practical to improve from this autonomously collected data without any human annotations. We carry out extensive experiments in the real world to demonstrate the effectiveness of our approach, and find that in a suite of unseen environments, the robot policy can be improved significantly with autonomously collected data. We open-source the code for our semantic autonomous improvement pipeline, as well as our autonomous dataset of 30.5K trajectories collected across five tabletop environments.

In the world of driverless cars, the one thing we hear about time and time again is safety. And that comes down to testing. Even though we may take it for granted once we've got our license, driving is an incredibly complex skill. For vehicles to manage the many processes involved in driving autonomously they need to be exposed to all imaginable scenarios and be programmed to deal with the outcomes. That's precisely why some autonomous test models drive as many as 20 million miles a day in simulated environments.

At some point in the not-too-distant future, artificial intelligence (AI) will drive our cars, write our programming code, and optimize how we do business. Data centers, too, will be unable to escape this trend. Thanks to machine learning technology, companies and data center operators will be able to coordinate and manage increasingly complex machines, infrastructures, and data more effectively than ever before, even as their numbers and data volumes continue to rise. Are completely autonomous, self-repairing data centers on the horizon? The data center is the backbone of the digital revolution.

In late November, Hewlett Packard Enterprise (HPE) announced the industry's first artificial intelligence (AI) recommendation engine designed to simplify and reduce the guesswork in managing infrastructure and improve application reliability. HPE InfoSight is an industry-leading predictive analytics platform that brings software-defined intelligence to the data center with the ability to predict and prevent infrastructure problems before they happen. Leveraging advanced machine learning, the new capabilities for HPE InfoSight pave the path toward an autonomous data center. HPE also announced the first release of HPE InfoSight for HPE 3PAR, which makes cross-stack analytics and global visibility available, and provides the foundation for predictive support. This declaration sets in motion the promise to extend machine learning across the HPE storage and server portfolio.

For decades, data and analytics have played an important role in our economy. The process of analyzing data, however, remains labor intensive. Even with the most advanced techniques, data scientists spend countless hours developing, testing and retooling analytic models one step at a time. Worse yet, most organizations cannot find enough data scientists to complete this labor-intensive work. The impact is that we have not yet fully realized the promise of continuous intelligence; until now.

The reality of a self-managing data center is getting closer with HPE's announcement last week of what it claims to be the first artificial intelligence (AI) predictive engine for trouble in the data center. HPE says next year it will offer an AI recommendation engine add-on that's designed to predict and stop storage- and general-infrastructure trouble before it starts. It's one of a number of autonomous data center components that we should expect to see soon from players. Other AI and machine learning systems geared towards data centers will be available from companies such as Litbit (which I wrote about in the summer) and Oracle, among others. "Infrastructure solutions should utilize data science and machine learning," HPE says in a white paper in which it attempts to explain why AI and machine learning are better at preventing downtime than humans. Currently, IT workers have to constantly carry out "intricate forensic work to unravel the maze of issues that impact data delivery to applications."

HPE is adding an AI-based recommendation engine to the InfoSight predictive analytics platform for flash storage, taking another step toward what it calls the autonomous data centre, where systems modify themselves to run more efficiently. The ultimate goal is to simplify and automate infrastructure management in order to cut operation expenses. HPE acquired InfoSight as part of its $1 billion deal earlier this year for Nimble Software, a maker of all-flash and hybrid flash storage products. Along with the announcement of the new recommendation engine, HPE Tuesday also said it is extending InfoSight to work with 3Par high-end storage technology it acquired in 2010. HPE says that is only the beginning of what it is doing to develop InfoSight's ability to monitor infrastructure, predict possible problems and recommend ways to enhance performance.

Machine learning has simplified the heavy lifting of collecting and analyzing data. Even with the most advanced techniques, data scientists spend countless hours developing, testing and retooling analytic models one step at a time. Worse yet, most organizations cannot find enough data scientists to complete this labor-intensive work. The impact is that we have not yet fully realized the promise of continuous intelligence; until now. The field of machine learning offers promises to streamline the application of analytics and create a new era of autonomous data. It adds massive efficiencies to the process by automating the construction of these models.