Diffusion-Based Attention Warping for Consistent 3D Scene Editing

Abstract

We present a novel method for 3D scene editing using diffusion models, designed to ensure view consistency and realism across perspectives. Our approach leverages attention features extracted from a single reference image to define the intended edits. These features are warped across multiple views by aligning them with scene geometry derived from Gaussian splatting depth estimates. Injecting these warped features into other viewpoints enables coherent propagation of edits, achieving high fidelity and spatial alignment in 3D space. Extensive evaluations demonstrate the effectiveness of our method in generating versatile edits of 3D scenes, significantly advancing the capabilities of scene manipulation compared to the existing methods.

Method

Our method uses 2D diffusion models with attention warping to ensure consistent 3D scene editing. By guiding the process with a single image, we optimize both computational efficiency and memory usage. Starting with a 3D scene represented by a Gaussian splatting model, we apply edits based on textual instructions and a reference image or depth map. We compute attention maps from the source view, then warp them to target views using depth and camera information, ensuring coherence across perspectives. Our approach goes beyond basic 3D scene editing—it allows users to select the style of the edits. Whether using a diffusion model or a non-AI-generated image, users have the flexibility to guide the scene modification in their preferred way. Below is an illustration of the method.

Results

Dinosaur

Face

Bear

Stump

BibTeX


      @misc{gomel2024diffusionbasedattentionwarpingconsistent,
        title={Diffusion-Based Attention Warping for Consistent 3D Scene Editing}, 
        author={Eyal Gomel and Lior Wolf},
        year={2024},
        eprint={2412.07984},
        archivePrefix={arXiv},
        primaryClass={cs.CV},
        url={https://arxiv.org/abs/2412.07984}, 
      }