diffusers / sdxl-instructpix2pix-768

SDXL InstructPix2Pix (768768)

Instruction fine-tuning of Stable Diffusion XL (SDXL) à la InstructPix2Pix . Some results below:

Edit instruction : "Turn sky into a cloudy one"

Edit instruction : "Make it a picasso painting"

Edit instruction : "make the person older"

Usage in 🧨 diffusers

Make sure to install the libraries first:

pip install accelerate transformers
pip install git+https://github.com/huggingface/diffusers

import torch
from diffusers import StableDiffusionXLInstructPix2PixPipeline
from diffusers.utils import load_image

resolution = 768
image = load_image(
    "https://hf.co/datasets/diffusers/diffusers-images-docs/resolve/main/mountain.png"
).resize((resolution, resolution))
edit_instruction = "Turn sky into a cloudy one"

pipe = StableDiffusionXLInstructPix2PixPipeline.from_pretrained(
    "diffusers/sdxl-instructpix2pix-768", torch_dtype=torch.float16
).to("cuda")

edited_image = pipe(
    prompt=edit_instruction,
    image=image,
    height=resolution,
    width=resolution,
    guidance_scale=3.0,
    image_guidance_scale=1.5,
    num_inference_steps=30,
).images[0]
edited_image.save("edited_image.png")

To know more, refer to the documentation .

🚨 Note that this checkpoint is experimental in nature and there's a lot of room for improvements. Please use the "Discussions" tab of this repository to open issues and discuss. 🚨

Training

We fine-tuned SDXL using the InstructPix2Pix training methodology for 15000 steps using a fixed learning rate of 5e-6 on an image resolution of 768x768.

Our training scripts and other utilities can be found here and they were built on top of our official training script .

Our training logs are available on Weights and Biases here . Refer to this link for details on all the hyperparameters.

Training data

We used this dataset: timbrooks/instructpix2pix-clip-filtered .

Compute

one 8xA100 machine

Batch size

Data parallel with a single gpu batch size of 8 for a total batch size of 32.

Mixed precision

FP16