TencentARC / Divot
Model's Last Updated: 12월 10 2024
Introduction of Divot
Model Details of Divot
Divot: Diffusion Powers Video Tokenizer for Comprehension and Generation
We introduce Divot, a Di ffusion-Powered V ide o T okenizer, which leverages the diffusion process for self-supervised video representation learning. We posit that if a video diffusion model can effectively de-noise video clips by taking the features of a video tokenizer as the condition, then the tokenizer has successfully captured robust spatial and temporal information. Additionally, the video diffusion model inherently functions as a de-tokenizer, decoding videos from their representations. Building upon the Divot tokenizer, we present Divot-LLM through video-to-text autoregression and text-to-video generation by modeling the distributions of continuous-valued Divot features with a Gaussian Mixture Model.
All models, training code and inference code are released!
TODOs
- Release the pretrained tokenizer and de-tokenizer of Divot.
- Release the pretrained and instruction tuned model of Divot-LLM.
- Release inference code of Divot.
- Release training and inference code of Divot-LLM.
- Release training code of Divot.
- Release de-tokenizer adaptation training code.
Introduction
We utilize the diffusion procedure to learn a video tokenizer in a self-supervised manner for unified comprehension and generation, where the spatiotemporal representations serve as the condition of a diffusion model to de-noise video clips. Additionally, the proxy diffusion model functions as a de-tokenizer to decode realistic video clips from the video representations.
After training the the Divot tokenizer, video features from the Divot tokenizer are fed into the LLM to perform next-word prediction for video comprehension, while learnable queries are input into the LLM to model the distributions of Divot features using a Gaussian Mixture Model (GMM) for video generation. During inference, video features are sampled from the predicted GMM distribution to decode videos using the de-tokenizer.
Usage
Dependencies
- Python >= 3.8 (Recommend to use Anaconda )
- PyTorch >=2.1.0
- NVIDIA GPU + CUDA
Installation
Clone the repo and install dependent packages
git clone https://github.com/TencentARC/Divot.git
cd Divot
pip install -r requirements.txt
Model Weights
We release the pretrained tokenizer and de-tokenizer, pre-trained and instruction-tuned Divot-LLM. Please download the checkpoints and save them under the folder
./pretrained
. For example,
./pretrained/Divot_tokenizer_detokenizer
.
You also need to download
Mistral-7B-Instruct-v0.1
and
CLIP-ViT-H-14-laion2B-s32B-b79K
, and save them under the folder
./pretrained
.
Inference
Video Reconstruction with Divot
python3 src/tools/eval_Divot_video_recon.py
Video Comprehension with Divot-LLM
python3 src/tools/eval_Divot_video_comp.py
Video Generation with Divot-LLM
python3 src/tools/eval_Divot_video_gen.py
Training
Pre-training
-
Download the checkpoints of pre-trained
Mistral-7B-Instruct-v0.1
and
CLIP-ViT-H-14-laion2B-s32B-b79K
, and save them under the folder
./pretrained. - Prepare the training data in the format of webdataset.
- Run the following script.
sh scripts/train_Divot_pretrain_comp_gen.sh
Instruction-tuning
-
Download the checkpoints of pre-trained Divot tokenizer and Divot-LLM in
Divot
, and save them under the folder
./pretrained. - Prepare the instruction data in the format of webdataset (for generation) and jsonl (for comprehension, where each line stores a dictionary used to specify the video_path, question, and answer).
- Run the following script.
### For video comprehension
sh scripts/train_Divot_sft_comp.sh
### For video generation
sh scripts/train_Divot_sft_gen.sh
Inference with your own model
- Obtain "pytorch_model.bin" with the following script.
cd train_output/sft_comp/checkpoint-xxxx
python3 zero_to_fp32.py . pytorch_model.bin
- Merge your trained lora with the original LLM model using the following script.
python3 src/tools/merge_agent_lora_weight.py
- Load your merged model in "mistral7b_merged_xxx" and and corresponding "agent" path, For example,
llm_cfg_path = 'configs/clm_models/mistral7b_merged_sft_comp.yaml'
agent_cfg_path = 'configs/clm_models/agent_7b_in64_out64_video_gmm_sft_comp.yaml'
License
Divot
is licensed under the Apache License Version 2.0 for academic purpose only except for the third-party components listed in
License
.
Acknowledge
Our code for Divot tokenizer and de-tokenizer is built upon DynamiCrafter . Thanks for their excellent work!
Runs of TencentARC Divot on huggingface.co
19
Total runs
0
24-hour runs
0
3-day runs
0
7-day runs
11
30-day runs
More Information About Divot huggingface.co Model
Url of Divot
Divot huggingface.co Url
Provider of Divot huggingface.co
Other API from TencentARC
Create photos, paintings and avatars for anyone in any style within seconds.
Total runs:
35.2K
Run Growth:
-43.4K
Growth Rate:
-124.12%
Total runs:
9.8K
Run Growth:
134
Growth Rate:
1.37%
Total runs:
6.8K
Run Growth:
1.1K
Growth Rate:
16.29%
Total runs:
6.7K
Run Growth:
363
Growth Rate:
5.55%
Total runs:
5.9K
Run Growth:
1.0K
Growth Rate:
17.41%
Total runs:
5.3K
Run Growth:
1.4K
Growth Rate:
27.11%
Total runs:
4.3K
Run Growth:
38
Growth Rate:
0.89%
Total runs:
2.6K
Run Growth:
327
Growth Rate:
12.57%
Total runs:
2.4K
Run Growth:
57
Growth Rate:
2.32%
Total runs:
2.4K
Run Growth:
152
Growth Rate:
6.31%
Total runs:
2.2K
Run Growth:
743
Growth Rate:
34.30%
Total runs:
2.0K
Run Growth:
210
Growth Rate:
10.38%
Total runs:
217
Run Growth:
-181
Growth Rate:
-82.27%
Total runs:
148
Run Growth:
-542
Growth Rate:
-354.25%
Total runs:
57
Run Growth:
35
Growth Rate:
61.40%
Total runs:
54
Run Growth:
-18
Growth Rate:
-35.29%
Total runs:
50
Run Growth:
29
Growth Rate:
60.42%
Total runs:
41
Run Growth:
12
Growth Rate:
30.77%
Total runs:
41
Run Growth:
-103
Growth Rate:
-264.10%
Total runs:
33
Run Growth:
-78
Growth Rate:
-243.75%
Total runs:
27
Run Growth:
9
Growth Rate:
40.91%
Total runs:
24
Run Growth:
3
Growth Rate:
12.50%
Total runs:
19
Run Growth:
3
Growth Rate:
15.79%
Total runs:
18
Run Growth:
9
Growth Rate:
50.00%
Total runs:
12
Run Growth:
2
Growth Rate:
16.67%
Total runs:
12
Run Growth:
2
Growth Rate:
16.67%
Total runs:
12
Run Growth:
2
Growth Rate:
16.67%
Total runs:
9
Run Growth:
2
Growth Rate:
22.22%
Total runs:
7
Run Growth:
-5
Growth Rate:
-71.43%
Total runs:
6
Run Growth:
-2
Growth Rate:
-33.33%
Total runs:
5
Run Growth:
-2
Growth Rate:
-40.00%
Total runs:
5
Run Growth:
-6
Growth Rate:
-120.00%
Total runs:
4
Run Growth:
4
Growth Rate:
100.00%
Total runs:
4
Run Growth:
4
Growth Rate:
100.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%