ai-forever / kandinsky-4-v2a

huggingface.co
Total runs: 13
24-hour runs: 2
7-day runs: -43
30-day runs: -41
Model's Last Updated: December 13 2024

Introduction of kandinsky-4-v2a

Model Details of kandinsky-4-v2a

Kandinsky-4-v2a: A Video to Audio pipeline





Description

Video to Audio pipeline consists of a visual encoder, a text encoder, UNet diffusion model to generate spectrogram and Griffin-lim algorithm to convert spectrogram into audio. Visual and text encoders share the same multimodal visual language decoder ( cogvlm2-video-llama3-chat ).

Our UNet diffusion model is a finetune of the music generation model riffusion . We made modifications in the architecture to condition on video frames and improve the synchronization between video and audio. Also, we replace the text encoder with the decoder of cogvlm2-video-llama3-chat .

image/png

Installation
git clone https://github.com/ai-forever/Kandinsky-4.git
cd Kandinsky-4
conda install -c conda-forge ffmpeg -y
pip install -r kandinsky4_video2audio/requirements.txt
pip install "git+https://github.com/facebookresearch/pytorchvideo.git"
Inference

Inference code for Video-to-Audio:

import torch
import torchvision

from kandinsky4_video2audio.video2audio_pipe import Video2AudioPipeline
from kandinsky4_video2audio.utils import load_video, create_video

device='cuda:0'

pipe = Video2AudioPipeline(
    "ai-forever/kandinsky-4-v2a",
    torch_dtype=torch.float16,
    device = device
)

video_path = 'assets/inputs/1.mp4'
video, _, fps = torchvision.io.read_video(video_path)

prompt="clean. clear. good quality."
negative_prompt = "hissing noise. drumming rythm. saying. poor quality."
video_input, video_complete, duration_sec = load_video(video, fps['video_fps'], num_frames=96, max_duration_sec=12)
    
out = pipe(
    video_input,
    prompt,
    negative_prompt=negative_prompt,
    duration_sec=duration_sec, 
)[0]

save_path = f'assets/outputs/1.mp4'
create_video(
    out, 
    video_complete, 
    display_video=True,
    save_path=save_path,
    device=device
)

Authors

Runs of ai-forever kandinsky-4-v2a on huggingface.co

13
Total runs
2
24-hour runs
1
3-day runs
-43
7-day runs
-41
30-day runs

More Information About kandinsky-4-v2a huggingface.co Model

More kandinsky-4-v2a license Visit here:

https://choosealicense.com/licenses/apache-2.0

kandinsky-4-v2a huggingface.co

kandinsky-4-v2a huggingface.co is an AI model on huggingface.co that provides kandinsky-4-v2a's model effect (), which can be used instantly with this ai-forever kandinsky-4-v2a model. huggingface.co supports a free trial of the kandinsky-4-v2a model, and also provides paid use of the kandinsky-4-v2a. Support call kandinsky-4-v2a model through api, including Node.js, Python, http.

kandinsky-4-v2a huggingface.co Url

https://huggingface.co/ai-forever/kandinsky-4-v2a

ai-forever kandinsky-4-v2a online free

kandinsky-4-v2a huggingface.co is an online trial and call api platform, which integrates kandinsky-4-v2a's modeling effects, including api services, and provides a free online trial of kandinsky-4-v2a, you can try kandinsky-4-v2a online for free by clicking the link below.

ai-forever kandinsky-4-v2a online free url in huggingface.co:

https://huggingface.co/ai-forever/kandinsky-4-v2a

kandinsky-4-v2a install

kandinsky-4-v2a is an open source model from GitHub that offers a free installation service, and any user can find kandinsky-4-v2a on GitHub to install. At the same time, huggingface.co provides the effect of kandinsky-4-v2a install, users can directly use kandinsky-4-v2a installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

kandinsky-4-v2a install url in huggingface.co:

https://huggingface.co/ai-forever/kandinsky-4-v2a

Url of kandinsky-4-v2a

kandinsky-4-v2a huggingface.co Url

Provider of kandinsky-4-v2a huggingface.co

ai-forever
ORGANIZATIONS

Other API from ai-forever

huggingface.co

Total runs: 525.5K
Run Growth: 507.4K
Growth Rate: 96.56%
Updated: November 03 2023
huggingface.co

Total runs: 10.6K
Run Growth: 1.5K
Growth Rate: 13.75%
Updated: December 05 2023
huggingface.co

Total runs: 8.2K
Run Growth: 5.1K
Growth Rate: 59.45%
Updated: December 29 2024
huggingface.co

Total runs: 5.9K
Run Growth: 3.5K
Growth Rate: 60.26%
Updated: December 11 2023
huggingface.co

Total runs: 2.3K
Run Growth: -408
Growth Rate: -17.86%
Updated: December 05 2023
huggingface.co

Total runs: 1.3K
Run Growth: -6.3K
Growth Rate: -493.39%
Updated: December 05 2023
huggingface.co

Total runs: 315
Run Growth: 158
Growth Rate: 50.16%
Updated: January 26 2023
huggingface.co

Total runs: 0
Run Growth: 0
Growth Rate: 0.00%
Updated: December 24 2021
huggingface.co

Total runs: 0
Run Growth: 0
Growth Rate: 0.00%
Updated: June 08 2023
huggingface.co

Total runs: 0
Run Growth: 0
Growth Rate: 0.00%
Updated: September 21 2021