Real-world evaluation of perception-based planning models for robotic systems, such as autonomous vehicles, can be safely and inexpensively conducted offline, i.e. by computing model prediction error over a pre-collected validation dataset with ground-truth annotations. However, extrapolating from offline model performance to online settings remains a challenge. In these settings, seemingly minor errors can compound and result in test-time infractions or collisions. This relationship is understudied, particularly across diverse closed-loop metrics and complex urban maneuvers. In this work, we revisit this undervalued question in policy evaluation through an extensive set of experiments across diverse conditions and metrics. Based on analysis in simulation, we find an even worse correlation between offline and online settings than reported by prior studies, casting doubts on the validity of current evaluation practices and metrics for driving policies. Next, we bridge the gap between offline and online evaluation. We investigate an offline metric based on epistemic uncertainty, which aims to capture events that are likely to cause errors in closed-loop settings. The resulting metric achieves over 13% improvement in correlation compared to previous offline metrics. We further validate the generalization of our findings beyond the simulation environment in real-world settings, where even greater gains are observed.

Existing evaluation paradigms for Autonomous Vehicles (AVs) face critical limitations. Real-world evaluation is often challenging due to safety concerns and a lack of reproducibility, whereas closed-loop simulation can face insufficient realism or high computational costs. Open-loop evaluation, while being efficient and data-driven, relies on metrics that generally overlook compounding errors. In this paper, we propose pseudo-simulation, a novel paradigm that addresses these limitations. Pseudo-simulation operates on real datasets, similar to open-loop evaluation, but augments them with synthetic observations generated prior to evaluation using 3D Gaussian Splatting. Our key idea is to approximate potential future states the AV might encounter by generating a diverse set of observations that vary in position, heading, and speed. Our method then assigns a higher importance to synthetic observations that best match the AV's likely behavior using a novel proximity-based weighting scheme. This enables evaluating error recovery and the mitigation of causal confusion, as in closed-loop benchmarks, without requiring sequential interactive simulation. We show that pseudo-simulation is better correlated with closed-loop simulations ($R^2=0.8$) than the best existing open-loop approach ($R^2=0.7$). We also establish a public leaderboard for the community to benchmark new methodologies with pseudo-simulation. Our code is available at https://github.com/autonomousvision/navsim.

Heuristic search-based planning techniques are commonly used for motion planning on discretized spaces. The performance of these algorithms is heavily affected by the resolution at which the search space is discretized. Typically a fixed resolution is chosen for a given domain. While a finer resolution allows for better maneuverability, it significantly increases the size of the state space, and hence demands more search efforts. On the contrary, a coarser resolution gives a fast exploratory behavior but compromises on maneuverability and the completeness of the search. To effectively leverage the advantages of both high and low resolution discretizations, we propose Multi-Resolution A* (MRA*) algorithm, that runs multiple weighted-A*(WA*) searches having different resolution levels simultaneously and combines the strengths of all of them. In addition to these searches, MRA* uses one anchor search to control expansions from these searches. We show that MRA* is bounded suboptimal with respect to the anchor resolution search space and resolution complete. We performed experiments on several motion planning domains including 2D, 3D grid planning and 7 DOF manipulation planning and compared our approach with several search-based and sampling-based baselines.

What does Magic Johnson and a twenty foot robot have in common? You guessed it, Automate 2017. While this might seem like an odd pairing, it accurately reflects the current state of the robotics industry. Already 2017 is on pace to beat last year's $19 billion investment record, with the recent announcements of Intel's $15B purchase of Mobileye; ABB's $2B acquisition of Bernecker & Rainer; and Ford's $1B investment in Argo AI. The excitement inside Chicago's McCormick Center on Wednesday was palpable, as Automate 2017 brought together not just established industrial leaders but also innovative startups at The Launchpad Competition.

A keen-eyed new robot looks poised to snag an important everyday warehouse job. Kinema Systems, a startup based in Menlo Park, California, has developed a robot capable of breaking down pallets of boxes no matter what size or shape they are or how they are packed together. This is a routine job at thousands of large stores, warehouses, and shipping companies--the aftermath of goods making their way through the supply line to your front door. The new robot uses a simple suction system to grab boxes, but it needs state-of-the-art computer vision and machine learning to figure out how to grab them. The machine does not need to be programmed at all--instead, it automatically calibrates itself and teaches itself how to break boxes down.

Today, Kinema Systems, a robotics startup based in Palo Alto, Calif., is coming out of stealth mode to announce Kinema Pick, which is "the world's first self-training, self-calibrating software solution for robotic depalletizing." I know, it sounds a little dry, but they have a convincingly cool demo, and we have lots of details on how the system works (and why it's important) from Kinema co-founder and CEO Sachin Chitta. Depalletizing is the task of picking up boxes of stuff off of shipping pallets and doing something with them. If this sounds like a task that should be easy and useful to automate with an industrial robot arm, that's because it is, with the caveat that it's only easy if you get the same pallets with the same boxes on them over and over again. E-commerce companies are getting pallets with all kinds of random boxes tightly jammed on there however they'll fit, which is too much variability for most robots to handle.