Kandinsky Video is a text-to-video generation model, which is based on the FusionFrames architecture, consisting of two main stages: keyframe generation and interpolation. Our approach for temporal conditioning allows us to generate videos with high-quality appearance, smoothness and dynamics.
Pipeline
The encoded text prompt enters the U-Net keyframe generation model with temporal layers or blocks, and then the sampled latent keyframes are sent to the latent interpolation model in such a way as to predict three interpolation frames between two keyframes. A temporal MoVQ-GAN decoder is used to get the final video result.
Architecture details
Text encoder (Flan-UL2) - 8.6B
Latent Diffusion U-Net3D - 4.0B
MoVQ encoder/decoder - 256M
How to use
Check our jupyter notebooks with examples in
./examples
folder
1. text2video
from video_kandinsky3 import get_T2V_pipeline
t2v_pipe = get_T2V_pipeline('cuda', fp16=True)
pfps = 'medium'# ['low', 'medium', 'high']
video = t2v_pipe(
'a red car is drifting on the mountain road, close view, fast movement',
width=640, height=384, fps=fps
)
Results
"A car moving on the road from the sea to the mountains"
"A red car drifting, 4k video"
"Chemistry laboratory, chemical explosion, 4k"
"Erupting volcano raw power, molten lava, and the forces of the Earth"
KandinskyVideo huggingface.co is an AI model on huggingface.co that provides KandinskyVideo's model effect (), which can be used instantly with this ai-forever KandinskyVideo model. huggingface.co supports a free trial of the KandinskyVideo model, and also provides paid use of the KandinskyVideo. Support call KandinskyVideo model through api, including Node.js, Python, http.
KandinskyVideo huggingface.co is an online trial and call api platform, which integrates KandinskyVideo's modeling effects, including api services, and provides a free online trial of KandinskyVideo, you can try KandinskyVideo online for free by clicking the link below.
ai-forever KandinskyVideo online free url in huggingface.co:
KandinskyVideo is an open source model from GitHub that offers a free installation service, and any user can find KandinskyVideo on GitHub to install. At the same time, huggingface.co provides the effect of KandinskyVideo install, users can directly use KandinskyVideo installed effect in huggingface.co for debugging and trial. It also supports api for free installation.