Goto

Collaborating Authors

 avatar


SRHand: Super-Resolving Hand Images and 3D Shapes via View/Pose-aware Neural Image Representations and Explicit 3DMeshes

Neural Information Processing Systems

Reconstructing detailed hand avatars plays a crucial role in various applications. While prior works have focused on capturing high-fidelity hand geometry, they heavily rely on high-resolution multi-view image inputs and struggle to generalize on low-resolution images. Multi-view image super-resolution methods have been proposed to enforce 3D view consistency. These methods, however, are limited to static objects/scenes with fixed resolutions and are not applicable to articulated deformable hands. In this paper, we propose SRHand (Super-Resolution Hand), the method for reconstructing detailed 3D geometry as well as textured images of hands from low-resolution images.


BecomingLit: Relightable Gaussian Avatars with Hybrid Neural Shading

Neural Information Processing Systems

We introduce BecomingLit, a novel method for reconstructing relightable, highresolution head avatars that can be rendered from novel viewpoints at interactive rates. Therefore, we propose a new low-cost light stage capture setup, tailored specifically towards capturing faces. Using this setup, we collect a novel dataset consisting of diverse multi-view sequences of numerous subjects under varying illumination conditions and facial expressions. By leveraging our new dataset, we introduce a new relightable avatar representation based on 3DGaussian primitives that we animate with a parametric head model and an expression-dependent dynamics module. We propose a new hybrid neural shading approach, combining a neural diffuse BRDF with an analytical specular term. Our method reconstructs disentangled materials from our dynamic light stage recordings and enables allfrequency relighting of our avatars with both point lights and environment maps. In addition, our avatars can easily be animated and controlled from monocular videos. We validate our approach in extensive experiments on our dataset, where we consistently outperform existing state-of-the-art methods in relighting and reenactment by a significant margin.


TGA: True-to-Geometry Avatar Dynamic Reconstruction

Neural Information Processing Systems

Recent advances in 3DGaussian Splatting (3DGS) have improved the visual fidelity of dynamic avatar reconstruction. However, existing methods often overlook the inherent chromatic similarity of human skin tones, leading to poor capture of intricate facial geometry under subtle appearance changes. This is caused by the affine approximation of Gaussian projection, which fails to be perspective-aware to depth-induced shear effects. To this end, we propose True-to-Geometry Avatar Dynamic Reconstruction (TGA), a perspective-aware 4DGaussian avatar framework that sensitively captures fine-grained facial variations for accurate 3D geometry reconstruction. Specifically, to enable color-sensitive and geometry-consistent Gaussian representations under dynamic conditions, we introduce the PerspectiveAware Gaussian Transformation that jointly models temporal deformations and spatial projection by integrating Jacobian-guided adaptive deformation into the homogeneous formulation. Furthermore, we develop Incremental BVHTree Pivoting to enable fast frame-by-frame mesh extraction for 4DGaussian representations. A dynamic Gaussian Bounding Volume Hierarchy (BVH) tree is used to model the topological relationships among points, where active ones are filtered out by BVH pivoting and subsequently re-triangulated for surface reconstruction. Extensive experiments demonstrate that TGA achieves superior geometric accuracy.



VASA-3D: Lifelike Audio-Driven Gaussian Head Avatars from a Single Image

Neural Information Processing Systems

We propose VASA-3D, an audio-driven, single-shot 3D head avatar generator. This research tackles two major challenges: capturing the subtle expression details present in real human faces, and reconstructing an intricate 3D head avatar from a single portrait image. To accurately model expression details, VASA-3D leverages the motion latent of VASA-1, a method that yields exceptional realism and vividness in 2D talking heads. A critical element of our work is translating this motion latent to 3D, which is accomplished by devising a 3D head model that is conditioned on the motion latent. Customization of this model to a single image is achieved through an optimization framework that employs numerous video frames of the reference head synthesized from the input image. The optimization takes various training losses robust to artifacts and limited pose coverage in the generated training data. Our experiment shows that VASA-3D produces realistic 3D talking heads that cannot be achieved by prior art, and it supports the online generation of 512x512 free-viewpoint videos at up to 75 FPS, facilitating more immersive engagements with lifelike 3D avatars.


MPMAvatar: Learning 3D Gaussian Avatars with Accurate and Robust Physics-Based Dynamics

Neural Information Processing Systems

While there has been significant progress in the field of 3D avatar creation from visual observations, modeling physically plausible dynamics of humans with loose garments remains a challenging problem. Although a few existing works address this problem by leveraging physical simulation, they suffer from limited accuracy or robustness to novel animation inputs. In this work, we present MPMAvatar, a framework for creating 3D human avatars from multi-view videos that supports highly realistic, robust animation, as well as photorealistic rendering from free viewpoints. For accurate and robust dynamics modeling, our key idea is to use a Material Point Method-based simulator, which we carefully tailor to model garments with complex deformations and contact with the underlying body by incorporating an anisotropic constitutive model and a novel collision handling algorithm. We combine this dynamics modeling scheme with our canonical avatar that can be rendered using 3D Gaussian Splatting with quasi-shadowing, enabling high-fidelity rendering for physically realistic animations. In our experiments, we demonstrate that MPMAvatar significantly outperforms the existing state-of-the-art physics-based avatar in terms of (1) dynamics modeling accuracy, (2) rendering accuracy, and (3) robustness and efficiency. Additionally, we present a novel application in which our avatar generalizes to unseen interactions in a zero-shot manner--which was not achievable with previous learning-based methods due to their limited simulation generalizability. Our code will be publicly available.


BecomingLit: Relightable Gaussian Avatars with Hybrid Neural Shading

Neural Information Processing Systems

We introduce, a novel method for reconstructing relightable, high-resolution head avatars that can be rendered from novel viewpoints at interactive rates. Therefore, we propose a new low-cost light stage capture setup, tailored specifically towards capturing faces. Using this setup, we collect a novel dataset consisting of diverse multi-view sequences of numerous subjects under varying illumination conditions and facial expressions. By leveraging our new dataset, we introduce a new relightable avatar representation based on 3D Gaussian primitives that we animate with a parametric head model and an expression-dependent dynamics module. We propose a new hybrid neural shading approach, combining a neural diffuse BRDF with an analytical specular term. Our method reconstructs disentangled materials from our dynamic light stage recordings and enables all-frequency relighting of our avatars with both point lights and environment maps. In addition, our avatars can easily be animated and controlled from monocular videos. We validate our approach in extensive experiments on our dataset, where we consistently outperform existing state-of-the-art methods in relighting and reenactment by a significant margin.


Cameras, Sensors, and 3D Body Scans: All the Tech Helping Eliminate Blown Calls

WIRED

Soccer officials already rely on cameras to see who's offside and who sent the ball out of bounds. But during this World Cup, refs will use digital twins of each player to view plays from every angle. At the 2026 World Cup, the refs on the field and the officials on the sidelines will be able to use an abundance of tech to help call penalties, spot offside violations, and make other consequential decisions. The video assistant referee system, known as VAR, and the semi-automated offside technology (SAOT) have been used in soccer for years. But the setup at this summer's World Cup represents some of the most advanced uses of adjudication tech to date--not just in soccer, but across all high-level sports.


AI-powered version of Ozzy to appear in city

BBC News

A new AI-powered avatar of Black Sabbath singer Ozzy Osbourne could make its first UK appearance in Birmingham. Osbourne's wife Sharon and son Jack announced plans for the hyper-real version of the Birmingham-born singer at an expo in the US last week. Talking to Ed James on BBC Radio WM, she said that plans for the avatar were brilliant. I've seen the tests that they've done of Ozzy and you can see every pore on his face, his beard's coming through, it's that detailed, she said. Osbourne died in July aged 76, less than three weeks after he had performed at Villa Park with Black Sabbath.


Actress sues Avatar director for 'theft' of facial features

BBC News

Film-maker James Cameron and Disney are being sued by an actress who has accused the director of using her likeness as the basis for one of the lead characters in his hit film series Avatar. German-born US actress Q'orianka Kilcher, who is of indigenous Peruvian descent, alleged that in 2005 - when she was 14 - Cameron extracted her facial features from a photograph of her portraying Pocahontas in another film, The New World. In court documents filed on Tuesday in California, her team claimed Cameron directed his design team to use it as the foundation for the character of Neytiri, depicted on screen by Zoe Saldaña. BBC News has contacted Cameron and Disney for a comment. The Avatar movies contain a hybrid of live-action performance mixed with computer-generated characters.