GaussEdit: Adaptive 3D Scene Editing with Text and Image Prompts

Abstract--This paper presents GaussEdit, a framework for adaptive 3D scene editing guided by text and image prompts. The first stage involves initializing the 3D Gaussians to ensure high-quality edits. The second stage employs an Adaptive Global-Local Optimization strategy to balance global scene coherence and detailed local edits and a category-guided regularization technique to alleviate the Janus problem. The final stage enhances the texture of the edited objects using a sophisticated image-to-image synthesis technique, ensuring that the results are visually realistic and align closely with the given prompts. Our experimental results demonstrate that GaussEdit surpasses existing methods in editing accuracy, visual fidelity, and processing speed. By successfully embedding user-specified concepts into 3D scenes, GaussEdit is a powerful tool for detailed and user-driven 3D scene editing, offering significant improvements over traditional methods. These methods can efficiently reconstruct the details of complex scenes from multi-view images while maintaining a high degree of realism by learning the geometry and texture information of scenes through deep learning models. Their impact extends to a range of downstream tasks, such as texture editing [10], [11], shape deformation [12], [13], scene decomposition [14], and stylization [15]. These technologies provide unprecedented flexibility and precision in scene manipulation, significantly advancing the development of digital content creation and virtual reality applications. With the rapid development of large-scale text-to-image (T2I) models, generating or editing scenes solely based on text prompts [16]-[20] has unlocked new possibilities for traditional methods that rely on complex drawing and modeling [10], [12].