Collaborating Authors


Articulated Object Interaction in Unknown Scenes with Whole-Body Mobile Manipulation Artificial Intelligence

A kitchen assistant needs to operate human-scale objects, such as cabinets and ovens, in unmapped environments with dynamic obstacles. Autonomous interactions in such real-world environments require integrating dexterous manipulation and fluid mobility. While mobile manipulators in different form-factors provide an extended workspace, their real-world adoption has been limited. This limitation is in part due to two main reasons: 1) inability to interact with unknown human-scale objects such as cabinets and ovens, and 2) inefficient coordination between the arm and the mobile base. Executing a high-level task for general objects requires a perceptual understanding of the object as well as adaptive whole-body control among dynamic obstacles. In this paper, we propose a two-stage architecture for autonomous interaction with large articulated objects in unknown environments. The first stage uses a learned model to estimate the articulated model of a target object from an RGB-D input and predicts an action-conditional sequence of states for interaction. The second stage comprises of a whole-body motion controller to manipulate the object along the generated kinematic plan. We show that our proposed pipeline can handle complicated static and dynamic kitchen settings. Moreover, we demonstrate that the proposed approach achieves better performance than commonly used control methods in mobile manipulation. For additional material, please check: .

Injecting Knowledge in Data-driven Vehicle Trajectory Predictors Artificial Intelligence

Vehicle trajectory prediction tasks have been commonly tackled from two distinct perspectives: either with knowledge-driven methods or more recently with data-driven ones. On the one hand, we can explicitly implement domain-knowledge or physical priors such as anticipating that vehicles will follow the middle of the roads. While this perspective leads to feasible outputs, it has limited performance due to the difficulty to hand-craft complex interactions in urban environments. On the other hand, recent works use data-driven approaches which can learn complex interactions from the data leading to superior performance. However, generalization, \textit{i.e.}, having accurate predictions on unseen data, is an issue leading to unrealistic outputs. In this paper, we propose to learn a "Realistic Residual Block" (RRB), which effectively connects these two perspectives. Our RRB takes any off-the-shelf knowledge-driven model and finds the required residuals to add to the knowledge-aware trajectory. Our proposed method outputs realistic predictions by confining the residual range and taking into account its uncertainty. We also constrain our output with Model Predictive Control (MPC) to satisfy kinematic constraints. Using a publicly available dataset, we show that our method outperforms previous works in terms of accuracy and generalization to new scenes. We will release our code and data split here:

STEP: Stochastic Traversability Evaluation and Planning for Safe Off-road Navigation Artificial Intelligence

Although ground robotic autonomy has gained widespread usage in structured and controlled environments, autonomy in unknown and off-road terrain remains a difficult problem. Extreme, off-road, and unstructured environments such as undeveloped wilderness, caves, and rubble pose unique and challenging problems for autonomous navigation. To tackle these problems we propose an approach for assessing traversability and planning a safe, feasible, and fast trajectory in real-time. Our approach, which we name STEP (Stochastic Traversability Evaluation and Planning), relies on: 1) rapid uncertainty-aware mapping and traversability evaluation, 2) tail risk assessment using the Conditional Value-at-Risk (CVaR), and 3) efficient risk and constraint-aware kinodynamic motion planning using sequential quadratic programming-based (SQP) model predictive control (MPC). We analyze our method in simulation and validate its efficacy on wheeled and legged robotic platforms exploring extreme terrains including an underground lava tube.

A trusty robot to carry farms into the future


Farming is a tough business. Global food demand is surging, with as many as 10 billion mouths to feed by 2050. At the same time, environmental challenges and labor limitations have made the future uncertain for agricultural managers. A new company called Future Acres proposed to enable farmers to do more with less through the power of robots. The company, helmed by CEO Suma Reddy, who previously served as COO and co-founder at Farmself and has held multiple roles and lead companies focused on the agtech space, has created an autonomous, electric agricultural robotic harvest companion named Carry to help farmers gather hand-picked crops faster and with less physical demand. Automation has been playing an increasingly large role in agriculture, and agricultural robots are widely expected to play a critical role in food production going forward.

A trusty robot to carry farms into the future


Farming is a tough business. Global food demand is surging, with as many as 10 billion mouths to feed by 2050. At the same time, environmental challenges and labor limitations have made the future uncertain for agricultural managers. A new company called Future Acres proposed to enable farmers to do more with less through the power of robots. The company, helmed by CEO Suma Reddy, who previously served as COO and co-founder at Farmself and has held multiple roles and lead companies focused on the agtech space, has created an autonomous, electric agricultural robotic harvest companion named Carry to help farmers gather hand-picked crops faster and with less physical demand. Automation has been playing an increasingly large role in agriculture, and agricultural robots are widely expected to play a critical role in food production going forward.

iX-BSP: Incremental Belief Space Planning Artificial Intelligence

Deciding what's next? is a fundamental problem in robotics and Artificial Intelligence. Under belief space planning (BSP), in a partially observable setting, it involves calculating the expected accumulated belief-dependent reward, where the expectation is with respect to all future measurements. Since solving this general un-approximated problem quickly becomes intractable, state of the art approaches turn to approximations while still calculating planning sessions from scratch. In this work we propose a novel paradigm, Incremental BSP (iX-BSP), based on the key insight that calculations across planning sessions are similar in nature and can be appropriately re-used. We calculate the expectation incrementally by utilizing Multiple Importance Sampling techniques for selective re-sampling and re-use of measurement from previous planning sessions. The formulation of our approach considers general distributions and accounts for data association aspects. We demonstrate how iX-BSP could benefit existing approximations of the general problem, introducing iML-BSP, which re-uses calculations across planning sessions under the common Maximum Likelihood assumption. We evaluate both methods and demonstrate a substantial reduction in computation time while statistically preserving accuracy. The evaluation includes both simulation and real-world experiments considering autonomous vision-based navigation and SLAM. As a further contribution, we introduce to iX-BSP the non-integral wildfire approximation, allowing one to trade accuracy for computational performance by averting from updating re-used beliefs when they are "close enough". We evaluate iX-BSP under wildfire demonstrating a substantial reduction in computation time while controlling the accuracy sacrifice. We also provide analytical and empirical bounds of the effect wildfire holds over the objective value.

Soft robots for ocean exploration and offshore operations: A perspective


Most of the ocean is unknown. Yet we know that the most challenging environments on the planet reside in it. Understanding the ocean in its totality is a key component for the sustainable development of human activities and for the mitigation of climate change, as proclaimed by the United Nations. We are glad to share our perspective about the role of soft robots in ocean exploration and offshore operations at the outset of the ocean decade (2021-2030). In this study of the Soft Systems Group (part of The School of Engineering at The University of Edinburgh), we focus on the two ends of the water column: the abyss and the surface.

MPC-MPNet: Model-Predictive Motion Planning Networks for Fast, Near-Optimal Planning under Kinodynamic Constraints Artificial Intelligence

Kinodynamic Motion Planning (KMP) is to find a robot motion subject to concurrent kinematics and dynamics constraints. To date, quite a few methods solve KMP problems and those that exist struggle to find near-optimal solutions and exhibit high computational complexity as the planning space dimensionality increases. To address these challenges, we present a scalable, imitation learning-based, Model-Predictive Motion Planning Networks framework that quickly finds near-optimal path solutions with worst-case theoretical guarantees under kinodynamic constraints for practical underactuated systems. Our framework introduces two algorithms built on a neural generator, discriminator, and a parallelizable Model Predictive Controller (MPC). The generator outputs various informed states towards the given target, and the discriminator selects the best possible subset from them for the extension. The MPC locally connects the selected informed states while satisfying the given constraints leading to feasible, near-optimal solutions. We evaluate our algorithms on a range of cluttered, kinodynamically constrained, and underactuated planning problems with results indicating significant improvements in computation times, path qualities, and success rates over existing methods.

A general framework for modeling and dynamic simulation of multibody systems using factor graphs Machine Learning

In this paper, we present a novel general framework grounded in the factor graph theory to solve kinematic and dynamic problems for multi-body systems. Although the motion of multi-body systems is considered to be a well-studied problem and various methods have been proposed for its solution, a unified approach providing an intuitive interpretation is still pursued. We describe how to build factor graphs to model and simulate multibody systems using both, independent and dependent coordinates. Then, batch optimization or a fixed-lag-smoother can be applied to solve the underlying optimization problem that results in a highly-sparse nonlinear minimization problem. The proposed framework has been tested in extensive simulations and validated against a commercial multibody software. We release a reference implementation as an open-source C++ library, based on the GTSAM framework, a well-known estimation library. Simulations of forward and inverse dynamics are presented, showing comparable accuracy with classical approaches. The proposed factor graph-based framework has the potential to be integrated into applications related with motion estimation and parameter identification of complex mechanical systems, ranging from mechanisms to vehicles, or robot manipulators.

Mars lander spies the planet's deep boundaries


Two years ago, NASA's InSight spacecraft alighted on the surface of Mars, aiming to glean clues to the planet's interior from the shaking of distant earthquakes and deep heat leaking from its soil. Mars, it turned out, had other ideas. Its sticky soil has thwarted InSight's heat probe, and in recent months howling winds have deafened its sensitive seismometers. Most mysteriously, the planet hasn't been rattled by the large marsquakes that could vividly illuminate its depths. Despite these hurdles, a precious clutch of small-but-clear quakes has enabled the InSight team to see hints of boundaries in the rock, tens and hundreds of kilometers below. They are clues to the planet's formation billions of years ago, when it was a hot ball of magma and heavier elements like iron sank to form a core, while lighter rocks rose up out of the mantle to form a capping lid of crust. The results, some debuting this month at an online meeting of the American Geophysical Union (AGU), show that the planet's crust is surprisingly thin, its mantle cooler than expected, and its large iron core still molten. The findings suggest that in its infancy, Mars efficiently shed heat—perhaps through a pattern of upwelling mantle rock and subducting crust similar to plate tectonics on Earth. “This may be evidence for a far more dynamic crust formation in Mars's early days,” says Stephen Mojzsis, a planetary scientist at the University of Colorado, Boulder, who is unaffiliated with the mission. The evidence has been hard won. Early in the mission, winds were quiet enough for InSight's seismometers, housed in a small dome placed on the surface, to hear a multitude of small quakes—nearly 500 in total. But since June, winds have shaken the surface strongly enough to smother all but a handful of new quakes. Yet frustratingly, the winds have not been strong enough to sweep away dust that is darkening the craft's solar panels and foreshadowing the mission's end sometime in the next few years. The seismometers are still running nonstop, but power constraints have forced the team to turn off a weather station when using the lander's robotic arm. “We are starting to feel the effects,” says Bruce Banerdt, InSight's principal investigator and a geophysicist at NASA's Jet Propulsion Laboratory. Meanwhile, the heat probe, about the length of a paper towel tube, is stuck in soil that compacted instead of crumbling as the rod tried to delve in. Mission engineers have used the robotic arm to push the probe down and scrape dirt on top. In the next month or two, they'll try once more to get the probe to burrow in, Banerdt says. “If that doesn't work, we'll call it a day and accept disappointment.” Perhaps the biggest disappointment is the lack of a marsquake larger than magnitude 4.5. The seismic waves of a large quake travel more deeply, reflecting off the core and mantle boundaries and even circling the planet on its surface. The multiple echoes of a large quake can enable just a single seismic station like InSight's to locate the quake's source. But above magnitude 4, Mars has been curiously silent—an apparent violation of the scaling laws that apply on Earth and the Moon, where 100 magnitude 3 events correspond to 10 magnitude 4 quakes, and so on. “That is a bit weird,” says Simon Stähler, a seismologist on the team from ETH Zurich. It could simply be that Mars's faults aren't big enough to sustain big strikes, or that its crust isn't brittle enough. But two moderate quakes, at magnitude 3.7 and 3.3, have been treasure troves for the mission. Traced to Cerberus Fossae, deep fissures in the crust 1600 kilometers east of the landing site that were suspected of being seismically active, the quakes sent a one-two punch of compressive pressure (P) waves, followed by sidewinding shear (S) waves, barreling toward the lander. Some of the waves were confined to the crust; others reflected off the top of the mantle. Offsets in the travel times of the P and S waves hint at the thickness of the crust and suggest distinct layers within it, Brigitte Knapmeyer-Endrun, a seismologist at the University of Cologne, said in an AGU presentation. The top layer may reflect material ground up in the planet's first billion years, a period of intense asteroid bombardment, says Steven Hauck, a planetary scientist at Case Western Reserve University. At 20 or 37 kilometers thick, depending on whether the reflections accurately trace the top of the mantle, the martian crust appears to be thinner than Earth's continental crust—a surprise. Researchers had thought that Mars, a smaller planet with less internal heat, would have built up a thicker crust, with heat escaping through limited conduction and bouts of volcanism. (Though Mars is volcanically dead today, giant volcanoes dot its surface.) A thin crust, however, might mean Mars was losing heat efficiently, recycling its early crust, rather than just building it up, perhaps through a rudimentary form of plate tectonics, Mojzsis says. A handful of distant quakes, originating some 4000 kilometers away, provided a further clue. Those waves traveled deep through the mantle and interacted with the mantle transition zone, a layer where pressure transforms the mineral olivine into wadsleyite. By analyzing the travel time of waves that passed above, below, and through the transition zone, the team located its depth—and found it shallower than expected, an indication of a cooler mantle. For the mantle to be this cool today suggests that convection—the swirling motions that, on Earth, drive tectonic plates and carry heat from the mantle to the surface—might have operated early on, says Quancheng Huang, a Ph.D. student at the University of Maryland, College Park, who presented some of the results at the AGU meeting. “Plate tectonics is a very effective way of cooling a planet.” A third science experiment aboard InSight probes deeper still, using tiny Doppler shifts in radio broadcasts sent from Earth to receivers on the probe to detect slight wobbles in the planet's spin. The size and consistency of the planet's iron core affect the wobbles, much as raw eggs spin differently from cooked ones. “We've had something like 350 hours of tracking,” says Véronique Dehant, a geophysicist at the Royal Observatory of Belgium. The preliminary results confirm that the core is liquid, with a radius compatible with previous estimates made by spacecraft measuring tiny variations in the planet's gravity, Dehant reports in her AGU poster. Those gravity estimates have found a core with a radius of about 1800 kilometers—taking up more than half the planet's diameter. Rebecca Fischer, a mineral physicist and modeler at Harvard University, isn't surprised at the signs of a liquid core. “It would be a pretty big surprise if it weren't,” she says. Sulfur and other elements mixed with the iron should help it to remain molten while cool, much as salt prevents icing. On Earth, convective motions in the molten outer core drive the magnetic dynamo. But on Mars, those motions seem to have stopped long ago—and without a magnetic field, the planet's atmosphere was vulnerable to the Sun's cosmic rays and leached water to space. Banerdt hopes to sharpen this fuzzy picture of the planet's interior, and he thinks calmer winds will soon make that possible. After two Earth years, the probe's first martian year is ending, and the quiet of the mission's first months is returning. “We're looking forward to another whole pile of event detections,” Banerdt says. And though the planet has not cooperated so far, perhaps the Big One is poised to strike Mars like a gong—a reverberation that would at last make all clear.