Code: github.com/PSRahul/product_photography_with_lora_sd. The README has the generated images.
The goal was a pipeline that generates realistic product photoshoot visuals, detects whether a given product is in an image, and enhances only that product while leaving the rest of the scene alone. It has three steps.
1. Fine-tune Stable Diffusion with LoRA
Product photography is treated as an art style the model can learn. Fully fine-tuning Stable Diffusion is too slow on limited GPUs, so I used LoRA on Stable Diffusion 1.5.
- Data: 65 product photos from Unsplash, captioned automatically with BLIP. Every caption also gets an activation tag that switches the learned style on at inference time.
- Training: kohya-based notebooks on Google Colab; 6 repeats, batch size 2, 10 epochs (1,950 steps).
- Inference: the Stable Diffusion Web UI with the LoRA checkpoint and a product-photography prompt template.
2. Find the product
Segment Anything proposes foreground masks, and CLIP classifies each one against text prompts for the product classes (hand bag, shoe, car, watch). The script reports which products are present and keeps their masks. Images with no matching product, such as a photo of the moon, are skipped.
3. Enhance it
The product mask drives image-to-image inpainting, with a depth ControlNet so that the product’s position and the background stay the same while its appearance changes.