Babies are tremendous learning machines, and key advances for AI may soon be found in the architecture of their little brains. If you think an artificial intelligence model running on thousands of cutting-edge computer chips is smart, allow me to introduce you to the concept of a 1-year-old. OK, so babies might not be able to write computer programs, solve advanced math problems, or debate philosophical ideas. But unlike today's AI models, which consume an ocean's worth of training data and as much energy as a small country, babies learn to make sense of the world with amazing efficiency. They identify new objects after seeing them once or twice, and they learn through fleeting observation and physical interaction. When it comes to improving AI, babies--and the architecture of their brains--might hold crucial insights.

Embed to control: A locally linear307 latent dynamics model for control from raw images.

Given a visual scene, humans have strong intuitions about how a scene can evolve over time under given actions. The intuition, often termed visual intuitive physics, is a critical ability that allows us to make effective plans to manipulate the scene to achieve desired outcomes without relying on extensive trial and error. In this paper, we present a framework capable of learning 3D-grounded visual intuitive physics models from videos of complex scenes. Our method is composed of a conditional Neural Radiance Field (NeRF)-style visual frontend and a 3D point-based dynamics prediction backend, using which we can impose strong relational and structural inductive bias to capture the structure of the underlying environment. Unlike existing intuitive point-based dynamics works that rely on the supervision of dense point trajectory from simulators, we relax the requirements and only assume access to multi-view RGB images and (imperfect) instance masks acquired using color prior.

In this work, we propose a unified framework, called Visual Reasoning with Differentiable Physics (VRDP) 1, that can jointly learn visual concepts and infer physics models of objects and their interactions from videos and language. This is achieved by seamlessly integrating three components: a visual perception module, a concept learner, and a differentiable physics engine. The visual perception module parses each video frame into object-centric trajectories and represents them as latent scene representations. The concept learner grounds visual concepts (e.g., color, shape, and material) from these object-centric representations based on the language, thus providing prior knowledge for the physics engine. The differentiable physics model, implemented as an impulse-based differentiable rigid-body simulator, performs differentiable physical simulation based on the grounded concepts to infer physical properties, such as mass, restitution, and velocity, by fitting the simulated trajectories into the video observations. Consequently, these learned concepts and physical models can explain what we have seen and imagine what is about to happen in future and counterfactual scenarios.

Neural Information Processing Systems http://nips.cc/

Neural Information Processing Systems http://nips.cc/

To decouple the learning of underlying scene geometry from dynamic motion, we represent the scene as a time-invariantsigneddistance function (SDF)whichservesasareference frame, along with a time-conditioned deformation field.

Neural Information Processing Systems http://nips.cc/