Code: github.com/PSRahul/image-editor-stable-diffusion. The README has the input and output images.
Given a photo and an object class such as “chair”, the task was to rotate that object while keeping the rest of the scene. It assumes one instance of a common object class per image.
Pipeline
- Mask the object. Language Segment-Anything combines GroundingDINO for open-set detection from a text prompt with Segment Anything for the mask.
- Remove it. The Stable Diffusion inpainting checkpoint fills the masked region with background.
- Render a new view. Zero-1-to-3 synthesises the cropped object from new azimuth and polar angles, for example rotated 45° in azimuth.
- Put it back. The novel view is resized and overlaid on the inpainted image, centred on the original bounding box.
What didn’t work
Inpainting with a plain “background” prompt worked for the chair image but not for the other samples. Zero-1-to-3 wouldn’t run locally in the time available, so the novel view came from a hosted Hugging Face Space. Reconciling CUDA and package versions across several research repositories took a lot of the time, and in the end the masking step ran on CPU.