stabilityai / stable-diffusion-3.5-controlnets

huggingface.co
Total runs: 73
24-hour runs: 4
7-day runs: 22
30-day runs: 25
Model's Last Updated: December 11 2024
text-to-image

Introduction of stable-diffusion-3.5-controlnets

Model Details of stable-diffusion-3.5-controlnets

Stable Diffusion 3.5 ControlNets

ControlNet Demo Image

Model

This repository provides a number of ControlNet models trained for use with Stable Diffusion 3.5 Large .

The following control types are available:

  • Canny - Use a Canny edge map to guide the structure of the generated image. This is especially useful for illustrations, but works with all styles.
  • Depth - use a depth map, generated by DepthFM, to guide generation. Some example use cases include generating architectural renderings, or texturing 3D assets.
  • Blur - can be used to perform extremely high fidelity upscaling. A common use case is to tile an input image, apply the ControlNet to each tile, and merge the tiles to produce a higher resolution image. A more in-depth description of this use case is here .
    • For Comfy users, this extension provides support.
    • We recommend tiling the image at a tile size between 128 and 512.

All currently released ControlNets are compatible only with Stable Diffusion 3.5 Large (8b).

Additional ControlNet models, including 2B versions of the variants above, and multiple other control types, will be added to this repository in the future.

Please note: This model is released under the Stability Community License . Visit Stability AI to learn or contact us for commercial licensing details.

License

Here are the key components of the license:

  • Free for non-commercial use: Individuals and organizations can use the model free of charge for non-commercial use, including scientific research.
  • Free for commercial use (up to $1M in annual revenue): Startups, small to medium-sized businesses, and creators can use the model for commercial purposes at no cost, as long as their total annual revenue is less than $1M.
  • Ownership of outputs: Retain ownership of the media generated without restrictive licensing implications.

For organizations with annual revenue more than $1M, please contact us here to inquire about an Enterprise License.

Usage

For local or self-hosted use, we recommend ComfyUI for node-based UI inference, or the standalone SD3.5 repo for programmatic use.

Support in 🧨 Diffusers is planned, and will be available soon.

Usage in ComfyUI

Please see the ComfyUI announcement blog post for details on usage within Comfy, including example workflows.

Usage in SD3.5 Standalone Repo

Install the repo:

git clone [email protected]:Stability-AI/sd3.5.git
pip install -r requirements.txt

Then, download the models and sample images like so:

input/canny.png
models/clip_g.safetensors
models/clip_l.safetensors
models/t5xxl.safetensors
models/sd3.5_large.safetensors
models/sd3.5_large_controlnet_canny.safetensors

and then you can run

python sd3_infer.py --controlnet_ckpt models/sd3.5_large_controlnet_canny.safetensors --controlnet_cond_image input/canny.png --prompt "An adorable fluffy pastel creature"

Which should give you an image like below:

An adorable fluffy pastel creature

The conditioning image should already be preprocessed before being used as input to the standalone repo; sd3.5 does not implement the preprocessing code below.

Preprocessing

Below are code snippets for preprocessing the various control image types.

Canny
import torchvision.transforms.functional as F
# assuming img is a PIL image
img = F.to_tensor(img)
img = cv2.cvtColor(img.transpose(1, 2, 0), cv2.COLOR_RGB2GRAY)
img = cv2.Canny(img, 100, 200)
Blur
import torchvision.transforms as transforms
# assuming img is a PIL image
gaussian_blur = transforms.GaussianBlur(kernel_size=50)
blurred_image = gaussian_blur(img)
Depth
# install depthfm from https://github.com/CompVis/depth-fm
import torchvision.transforms as transforms
from depthfm.dfm import DepthFM
depthfm_model = DepthFM(ckpt_path=checkpoint_path)
depthfm_model.eval()

# assuming img is a PIL image
img = F.to_tensor(img)
c, h, w = img.shape
img = F.interpolate(img, (512, 512), mode='bilinear', align_corners=False)
with torch.no_grad():
  img = self.depthfm_model(img, num_steps=2, ensemble_size=4)
img = F.interpolate(img, (h, w), mode='bilinear', align_corners=False)
Tips
  • We recommend starting with a ControlNet strength of 0.7-0.8, and adjusting as needed.
  • Euler sampler and a slightly higher step count (50-60) gives best results, especially with Canny.
  • Pass --text_encoder_device <device_name> to load the text encoders directly to VRAM, which can speed up the full inference loop at the cost of extra VRAM usage.
Uses

All uses of the model must be in accordance with our Acceptable Use Policy .

Out-of-Scope Uses

The model was not trained to be factual or true representations of people or events. As such, using the model to generate such content is out-of-scope of the abilities of this model.

Training Data and Strategy

These models were trained on a wide variety of data, including synthetic data and filtered publicly available data.

Safety

We believe in safe, responsible AI practices and take deliberate measures to ensure Integrity starts at the early stages of development. This means we have taken and continue to take reasonable steps to prevent the misuse of Stable Diffusion 3.5 by bad actors. For more information about our approach to Safety please visit our Safety page .

Integrity Evaluation

Our integrity evaluation methods include structured evaluations and red-teaming testing for certain harms. Testing was conducted primarily in English and may not cover all possible harms.

Risks identified and mitigations:
  • Harmful content: We have implemented safeguards that attempt to strike the right balance between usefulness and preventing harm. However, this does not guarantee that all possible harmful content has been removed. All developers and deployers should exercise caution and implement content safety guardrails based on their specific product policies and application use cases.
  • Misuse: Technical limitations and developer and end-user education can help mitigate against malicious applications of models. All users are required to adhere to our Acceptable Use Policy, including when applying fine-tuning and prompt engineering mechanisms. Please reference the Stability AI Acceptable Use Policy for information on violative uses of our products.
  • Privacy violations: Developers and deployers are encouraged to adhere to privacy regulations with techniques that respect data privacy.
Acknowledgements
  • Lvmin Zhang, Anyi Rao, and Maneesh Agrawala, authors of the original ControlNet paper .
  • Lvmin Zhang, who also developed the Tile ControlNet , which inspired the Blur ControlNet.
  • Diffusers library authors, whose code was referenced during development.
  • InstantX team, whose Flux and SD3 ControlNets were also referenced during training.
  • All early testers and raters of the models, and the Stability AI team.
Contact

Please report any issues with the model or contact us:

Runs of stabilityai stable-diffusion-3.5-controlnets on huggingface.co

73
Total runs
4
24-hour runs
11
3-day runs
22
7-day runs
25
30-day runs

More Information About stable-diffusion-3.5-controlnets huggingface.co Model

More stable-diffusion-3.5-controlnets license Visit here:

https://choosealicense.com/licenses/stabilityai-ai-community

stable-diffusion-3.5-controlnets huggingface.co

stable-diffusion-3.5-controlnets huggingface.co is an AI model on huggingface.co that provides stable-diffusion-3.5-controlnets's model effect (), which can be used instantly with this stabilityai stable-diffusion-3.5-controlnets model. huggingface.co supports a free trial of the stable-diffusion-3.5-controlnets model, and also provides paid use of the stable-diffusion-3.5-controlnets. Support call stable-diffusion-3.5-controlnets model through api, including Node.js, Python, http.

stable-diffusion-3.5-controlnets huggingface.co Url

https://huggingface.co/stabilityai/stable-diffusion-3.5-controlnets

stabilityai stable-diffusion-3.5-controlnets online free

stable-diffusion-3.5-controlnets huggingface.co is an online trial and call api platform, which integrates stable-diffusion-3.5-controlnets's modeling effects, including api services, and provides a free online trial of stable-diffusion-3.5-controlnets, you can try stable-diffusion-3.5-controlnets online for free by clicking the link below.

stabilityai stable-diffusion-3.5-controlnets online free url in huggingface.co:

https://huggingface.co/stabilityai/stable-diffusion-3.5-controlnets

stable-diffusion-3.5-controlnets install

stable-diffusion-3.5-controlnets is an open source model from GitHub that offers a free installation service, and any user can find stable-diffusion-3.5-controlnets on GitHub to install. At the same time, huggingface.co provides the effect of stable-diffusion-3.5-controlnets install, users can directly use stable-diffusion-3.5-controlnets installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

stable-diffusion-3.5-controlnets install url in huggingface.co:

https://huggingface.co/stabilityai/stable-diffusion-3.5-controlnets

Url of stable-diffusion-3.5-controlnets

stable-diffusion-3.5-controlnets huggingface.co Url

Provider of stable-diffusion-3.5-controlnets huggingface.co

stabilityai
ORGANIZATIONS

Other API from stabilityai

huggingface.co

Total runs: 143.2K
Run Growth: 8.3K
Growth Rate: 5.81%
Updated: August 04 2023
huggingface.co

Total runs: 137.2K
Run Growth: 16.4K
Growth Rate: 11.93%
Updated: July 10 2024
huggingface.co

Total runs: 34.3K
Run Growth: 3.4K
Growth Rate: 10.04%
Updated: August 09 2024
huggingface.co

Total runs: 378
Run Growth: -97.9K
Growth Rate: -25898.41%
Updated: August 03 2024
huggingface.co

Total runs: 0
Run Growth: 0
Growth Rate: 0.00%
Updated: July 10 2024
huggingface.co

Total runs: 0
Run Growth: 0
Growth Rate: 0.00%
Updated: April 14 2024