stabilityai / sd-vae-ft-ema

huggingface.co
Total runs: 16.8K
24-hour runs: 0
7-day runs: 2.8K
30-day runs: 5.7K
Model's Last Updated: June 05 2023

Introduction of sd-vae-ft-ema

Model Details of sd-vae-ft-ema

Improved Autoencoders

Utilizing

These weights are intended to be used with the 🧨 diffusers library . If you are looking for the model to use with the original CompVis Stable Diffusion codebase , come here .

How to use with 🧨 diffusers

You can integrate this fine-tuned VAE decoder to your existing diffusers workflows, by including a vae argument to the StableDiffusionPipeline

from diffusers.models import AutoencoderKL
from diffusers import StableDiffusionPipeline

model = "CompVis/stable-diffusion-v1-4"
vae = AutoencoderKL.from_pretrained("stabilityai/sd-vae-ft-ema")
pipe = StableDiffusionPipeline.from_pretrained(model, vae=vae)
Decoder Finetuning

We publish two kl-f8 autoencoder versions, finetuned from the original kl-f8 autoencoder on a 1:1 ratio of LAION-Aesthetics and LAION-Humans, an unreleased subset containing only SFW images of humans. The intent was to fine-tune on the Stable Diffusion training set (the autoencoder was originally trained on OpenImages) but also enrich the dataset with images of humans to improve the reconstruction of faces. The first, ft-EMA , was resumed from the original checkpoint, trained for 313198 steps and uses EMA weights. It uses the same loss configuration as the original checkpoint (L1 + LPIPS). The second, ft-MSE , was resumed from ft-EMA and uses EMA weights and was trained for another 280k steps using a different loss, with more emphasis on MSE reconstruction (MSE + 0.1 * LPIPS). It produces somewhat ``smoother'' outputs. The batch size for both versions was 192 (16 A100s, batch size 12 per GPU). To keep compatibility with existing models, only the decoder part was finetuned; the checkpoints can be used as a drop-in replacement for the existing autoencoder.

Original kl-f8 VAE vs f8-ft-EMA vs f8-ft-MSE

Evaluation
COCO 2017 (256x256, val, 5000 images)
Model train steps rFID PSNR SSIM PSIM Link Comments
original 246803 4.99 23.4 +/- 3.8 0.69 +/- 0.14 1.01 +/- 0.28 https://ommer-lab.com/files/latent-diffusion/kl-f8.zip as used in SD
ft-EMA 560001 4.42 23.8 +/- 3.9 0.69 +/- 0.13 0.96 +/- 0.27 https://huggingface.co/stabilityai/sd-vae-ft-ema-original/resolve/main/vae-ft-ema-560000-ema-pruned.ckpt slightly better overall, with EMA
ft-MSE 840001 4.70 24.5 +/- 3.7 0.71 +/- 0.13 0.92 +/- 0.27 https://huggingface.co/stabilityai/sd-vae-ft-mse-original/resolve/main/vae-ft-mse-840000-ema-pruned.ckpt resumed with EMA from ft-EMA, emphasis on MSE (rec. loss = MSE + 0.1 * LPIPS), smoother outputs
LAION-Aesthetics 5+ (256x256, subset, 10000 images)
Model train steps rFID PSNR SSIM PSIM Link Comments
original 246803 2.61 26.0 +/- 4.4 0.81 +/- 0.12 0.75 +/- 0.36 https://ommer-lab.com/files/latent-diffusion/kl-f8.zip as used in SD
ft-EMA 560001 1.77 26.7 +/- 4.8 0.82 +/- 0.12 0.67 +/- 0.34 https://huggingface.co/stabilityai/sd-vae-ft-ema-original/resolve/main/vae-ft-ema-560000-ema-pruned.ckpt slightly better overall, with EMA
ft-MSE 840001 1.88 27.3 +/- 4.7 0.83 +/- 0.11 0.65 +/- 0.34 https://huggingface.co/stabilityai/sd-vae-ft-mse-original/resolve/main/vae-ft-mse-840000-ema-pruned.ckpt resumed with EMA from ft-EMA, emphasis on MSE (rec. loss = MSE + 0.1 * LPIPS), smoother outputs
Visual

Visualization of reconstructions on 256x256 images from the COCO2017 validation dataset.


256x256: ft-EMA (left), ft-MSE (middle), original (right)

Runs of stabilityai sd-vae-ft-ema on huggingface.co

16.8K
Total runs
0
24-hour runs
1.3K
3-day runs
2.8K
7-day runs
5.7K
30-day runs

More Information About sd-vae-ft-ema huggingface.co Model

More sd-vae-ft-ema license Visit here:

https://choosealicense.com/licenses/mit

sd-vae-ft-ema huggingface.co

sd-vae-ft-ema huggingface.co is an AI model on huggingface.co that provides sd-vae-ft-ema's model effect (), which can be used instantly with this stabilityai sd-vae-ft-ema model. huggingface.co supports a free trial of the sd-vae-ft-ema model, and also provides paid use of the sd-vae-ft-ema. Support call sd-vae-ft-ema model through api, including Node.js, Python, http.

stabilityai sd-vae-ft-ema online free

sd-vae-ft-ema huggingface.co is an online trial and call api platform, which integrates sd-vae-ft-ema's modeling effects, including api services, and provides a free online trial of sd-vae-ft-ema, you can try sd-vae-ft-ema online for free by clicking the link below.

stabilityai sd-vae-ft-ema online free url in huggingface.co:

https://huggingface.co/stabilityai/sd-vae-ft-ema

sd-vae-ft-ema install

sd-vae-ft-ema is an open source model from GitHub that offers a free installation service, and any user can find sd-vae-ft-ema on GitHub to install. At the same time, huggingface.co provides the effect of sd-vae-ft-ema install, users can directly use sd-vae-ft-ema installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

sd-vae-ft-ema install url in huggingface.co:

https://huggingface.co/stabilityai/sd-vae-ft-ema

Url of sd-vae-ft-ema

Provider of sd-vae-ft-ema huggingface.co

stabilityai
ORGANIZATIONS

Other API from stabilityai

huggingface.co

Total runs: 143.2K
Run Growth: 8.3K
Growth Rate: 5.81%
Updated: August 04 2023
huggingface.co

Total runs: 137.2K
Run Growth: 16.4K
Growth Rate: 11.93%
Updated: July 10 2024
huggingface.co

Total runs: 34.3K
Run Growth: 3.4K
Growth Rate: 10.04%
Updated: August 09 2024
huggingface.co

Total runs: 378
Run Growth: -97.9K
Growth Rate: -25898.41%
Updated: August 03 2024
huggingface.co

Total runs: 0
Run Growth: 0
Growth Rate: 0.00%
Updated: July 10 2024
huggingface.co

Total runs: 0
Run Growth: 0
Growth Rate: 0.00%
Updated: April 13 2024