We present Kandinsky 3.1, the follow-up to the Kandinsky 3.0 model, a large-scale text-to-image generation model based on latent diffusion, continuing the series of text-to-image Kandinsky models and reflecting our progress to achieve higher quality and realism of image generation, which we have enhanced and enriched with a variety of useful features and modes to give users more opportunities to fully utilise the power of our new model.
Kandinsky Flash (Kandinsky 3.0 Refiner)
Diffusion models have problems with fast image generation. To address this problem, we trained a Kandinksy Flash model based on the
Adversarial Diffusion Distillation
approach with some modifications: we trained the model on latents, which reduced the memory overhead and removed distillation loss as it did not affect the training. Also, we applied Kandinsky Flash model to images generated from Kandinsky 3.0 to improve visual quality of generated images.
Architecture
For training Kandinsky Flash we used the following architecture of discriminator. It is the half of Kandinsky 3.0 U-Net encoder with additional head predictions.
How to use:
Check our jupyter notebooks with examples in
./examples
folder
from kandinsky3 import get_T2I_Flash_pipeline
device_map = torch.device('cuda:0')
dtype_map = {
'unet': torch.float32,
'text_encoder': torch.float16,
'movq': torch.float32,
}
t2i_pipe = get_T2I_Flash_pipeline(
device_map, dtype_map
)
res = t2i_pipe("A cute corgi lives in a house made out of sushi.")
Kandinsky Inpainting
Also, we released a newer version of inpainting model, which we additionally trained the model on the object detection dataset. This allowed to get more stable generation of objects. The new weights are available at
ai-forever/Kandinsky3.1
. Check the usage
example
.
Prompt beautification
Prompt plays crucial role in text-to-image generation. So, in Kandinsky 3.1 we decided to use language model for making prompt better. We used Intel's
neural-chat-7b-v3-1
with the following system prompt as the LLM:
### System: You are a prompt engineer. Your mission is to expand prompts written by user. You should provide the best prompt for text to image generation in English.
### User:
{prompt}
### Assistant:
{answer of the model}
Kandinsky 3.0 is an open-source text-to-image diffusion model built upon the Kandinsky2-x model family. In comparison to its predecessors, Kandinsky 3.0 incorporates more data and specifically related to Russian culture, which allows to generate pictures related to Russin culture. Furthermore, enhancements have been made to the text understanding and visual quality of the model, achieved by increasing the size of the text encoder and Diffusion U-Net models, respectively.
For more information: details of training, example of generations check out our
post
. The english version will be released in a couple of days.
Architecture details:
Architecture consists of three parts:
Text encoder Flan-UL2 (encoder part) - 8.6B
Latent Diffusion U-Net - 3B
MoVQ encoder/decoder - 267M
Models
We release our two models:
Base
: Base text-to-image diffusion model. This model was trained over 2M steps on 400 A100
Inpainting
: Inpainting version of the model. The model was initialized from final checkpoint of base model and trained 250k steps on 300 A100.
Installing
To install repo first one need to create conda environment:
The exact dependencies is got using
pip freeze
and can be found in
exact_requirements.txt
How to use:
Check our jupyter notebooks with examples in
./examples
folder
1. text2image
import sys
sys.path.append('..')
import torch
from kandinsky3 import get_T2I_pipeline
device_map = torch.device('cuda:0')
dtype_map = {
'unet': torch.float32,
'text_encoder': torch.float16,
'movq': torch.float32,
}
t2i_pipe = get_T2I_pipeline(
device_map, dtype_map,
)
res = t2i_pipe("A cute corgi lives in a house made out of sushi.")
res[0]
2. inpainting
from kandinsky3 import get_inpainting_pipeline
device_map = torch.device('cuda:0')
dtype_map = {
'unet': torch.float16,
'text_encoder': torch.float16,
'movq': torch.float32,
}
pipe = get_inpainting_pipeline(
device_map, dtype_map,
)
image = ... # PIL Image
mask = ... # Numpy array (HxW). Set 1 where image should be masked
image = inp_pipe( "A cute corgi lives in a house made out of sushi.", image, mask)
Examples of generations
"A beautiful landscape outdoors scene in the crochet knitting art style, drawing in style by Alfons Mucha"
"a yellow house at the edge of the danish fjord, in the style of eiko ojala, ingrid baars, ad posters, mountainous vistas, george ault, realistic details, dark white and dark gray, 4k"
"dragon fruit head, upper body, realistic, illustration by Joshua Hoffine Norman Rockwell, scary, creepy, biohacking, futurism, Zaha Hadid style"
"Amazing playful nice cute strawberry character, dynamic poze, surreal fantazy garden background, gorgeous masterpice, award winning photo, soft natural lighting, 3d, Blender, Octane render, tilt - shift, deep field, colorful, I can't believe how beautiful this is, colorful, cute and sweet baby - loved photo"
"beautiful fairy-tale desert, in the sky a wave of sand merges with the milky way, stars, cosmism, digital art, 8k"
"Car, mustang, movie, person, poster, car cover, person, in the style of alessandro gottardo, gold and cyan, gerald harvey jones, reflections, highly detailed illustrations, industrial urban scenes""
"cloud in blue sky, a red lip, collage art, shuji terayama, dreamy objects, surreal, criterion collection, showa era, intricate details, mirror"
@misc{arkhipkin2023kandinsky,
title={Kandinsky 3.0 Technical Report},
author={Vladimir Arkhipkin and Andrei Filatov and Viacheslav Vasilev and Anastasia Maltseva and Said Azizov and Igor Pavlov and Julia Agafonova and Andrey Kuznetsov and Denis Dimitrov},
year={2023},
eprint={2312.03511},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
Runs of ai-forever Kandinsky3.1 on huggingface.co
0
Total runs
0
24-hour runs
0
3-day runs
0
7-day runs
0
30-day runs
More Information About Kandinsky3.1 huggingface.co Model
Kandinsky3.1 huggingface.co is an AI model on huggingface.co that provides Kandinsky3.1's model effect (), which can be used instantly with this ai-forever Kandinsky3.1 model. huggingface.co supports a free trial of the Kandinsky3.1 model, and also provides paid use of the Kandinsky3.1. Support call Kandinsky3.1 model through api, including Node.js, Python, http.
Kandinsky3.1 huggingface.co is an online trial and call api platform, which integrates Kandinsky3.1's modeling effects, including api services, and provides a free online trial of Kandinsky3.1, you can try Kandinsky3.1 online for free by clicking the link below.
ai-forever Kandinsky3.1 online free url in huggingface.co:
Kandinsky3.1 is an open source model from GitHub that offers a free installation service, and any user can find Kandinsky3.1 on GitHub to install. At the same time, huggingface.co provides the effect of Kandinsky3.1 install, users can directly use Kandinsky3.1 installed effect in huggingface.co for debugging and trial. It also supports api for free installation.