Total runs:
66.3K
Run Growth:
-691
Growth Rate:
-1.04%
TencentARC / Divot
Model's Last Updated: Tháng 12 10 2024
Introduction of Divot
Model Details of Divot
Divot: Diffusion Powers Video Tokenizer for Comprehension and Generation
We introduce Divot, a Di ffusion-Powered V ide o T okenizer, which leverages the diffusion process for self-supervised video representation learning. We posit that if a video diffusion model can effectively de-noise video clips by taking the features of a video tokenizer as the condition, then the tokenizer has successfully captured robust spatial and temporal information. Additionally, the video diffusion model inherently functions as a de-tokenizer, decoding videos from their representations. Building upon the Divot tokenizer, we present Divot-LLM through video-to-text autoregression and text-to-video generation by modeling the distributions of continuous-valued Divot features with a Gaussian Mixture Model.
All models, training code and inference code are released!
TODOs
- Release the pretrained tokenizer and de-tokenizer of Divot.
- Release the pretrained and instruction tuned model of Divot-LLM.
- Release inference code of Divot.
- Release training and inference code of Divot-LLM.
- Release training code of Divot.
- Release de-tokenizer adaptation training code.
Introduction
We utilize the diffusion procedure to learn a video tokenizer in a self-supervised manner for unified comprehension and generation, where the spatiotemporal representations serve as the condition of a diffusion model to de-noise video clips. Additionally, the proxy diffusion model functions as a de-tokenizer to decode realistic video clips from the video representations.
After training the the Divot tokenizer, video features from the Divot tokenizer are fed into the LLM to perform next-word prediction for video comprehension, while learnable queries are input into the LLM to model the distributions of Divot features using a Gaussian Mixture Model (GMM) for video generation. During inference, video features are sampled from the predicted GMM distribution to decode videos using the de-tokenizer.
Usage
Dependencies
- Python >= 3.8 (Recommend to use Anaconda )
- PyTorch >=2.1.0
- NVIDIA GPU + CUDA
Installation
Clone the repo and install dependent packages
git clone https://github.com/TencentARC/Divot.git
cd Divot
pip install -r requirements.txt
Model Weights
We release the pretrained tokenizer and de-tokenizer, pre-trained and instruction-tuned Divot-LLM. Please download the checkpoints and save them under the folder
./pretrained
. For example,
./pretrained/Divot_tokenizer_detokenizer
.
You also need to download
Mistral-7B-Instruct-v0.1
and
CLIP-ViT-H-14-laion2B-s32B-b79K
, and save them under the folder
./pretrained
.
Inference
Video Reconstruction with Divot
python3 src/tools/eval_Divot_video_recon.py
Video Comprehension with Divot-LLM
python3 src/tools/eval_Divot_video_comp.py
Video Generation with Divot-LLM
python3 src/tools/eval_Divot_video_gen.py
Training
Pre-training
-
Download the checkpoints of pre-trained
Mistral-7B-Instruct-v0.1
and
CLIP-ViT-H-14-laion2B-s32B-b79K
, and save them under the folder
./pretrained. - Prepare the training data in the format of webdataset.
- Run the following script.
sh scripts/train_Divot_pretrain_comp_gen.sh
Instruction-tuning
-
Download the checkpoints of pre-trained Divot tokenizer and Divot-LLM in
Divot
, and save them under the folder
./pretrained. - Prepare the instruction data in the format of webdataset (for generation) and jsonl (for comprehension, where each line stores a dictionary used to specify the video_path, question, and answer).
- Run the following script.
### For video comprehension
sh scripts/train_Divot_sft_comp.sh
### For video generation
sh scripts/train_Divot_sft_gen.sh
Inference with your own model
- Obtain "pytorch_model.bin" with the following script.
cd train_output/sft_comp/checkpoint-xxxx
python3 zero_to_fp32.py . pytorch_model.bin
- Merge your trained lora with the original LLM model using the following script.
python3 src/tools/merge_agent_lora_weight.py
- Load your merged model in "mistral7b_merged_xxx" and and corresponding "agent" path, For example,
llm_cfg_path = 'configs/clm_models/mistral7b_merged_sft_comp.yaml'
agent_cfg_path = 'configs/clm_models/agent_7b_in64_out64_video_gmm_sft_comp.yaml'
License
Divot
is licensed under the Apache License Version 2.0 for academic purpose only except for the third-party components listed in
License
.
Acknowledge
Our code for Divot tokenizer and de-tokenizer is built upon DynamiCrafter . Thanks for their excellent work!
Runs of TencentARC Divot on huggingface.co
19
Total runs
0
24-hour runs
0
3-day runs
0
7-day runs
11
30-day runs
More Information About Divot huggingface.co Model
Url of Divot
Divot huggingface.co Url
Provider of Divot huggingface.co
Other API from TencentARC
Updated: Tháng tư 11 2024
Create photos, paintings and avatars for anyone in any style within seconds.
Total runs:
35.0K
Run Growth:
-43.4K
Growth Rate:
-124.12%
Total runs:
30.5K
Run Growth:
6.6K
Growth Rate:
21.61%
Total runs:
9.8K
Run Growth:
134
Growth Rate:
1.37%
Total runs:
6.8K
Run Growth:
1.1K
Growth Rate:
16.29%
Total runs:
6.7K
Run Growth:
363
Growth Rate:
5.55%
Total runs:
5.9K
Run Growth:
1.0K
Growth Rate:
17.41%
Total runs:
5.3K
Run Growth:
1.4K
Growth Rate:
27.11%
Total runs:
4.3K
Run Growth:
38
Growth Rate:
0.89%
Total runs:
2.6K
Run Growth:
327
Growth Rate:
12.57%
Total runs:
2.4K
Run Growth:
57
Growth Rate:
2.32%
Total runs:
2.4K
Run Growth:
152
Growth Rate:
6.31%
Total runs:
2.3K
Run Growth:
962
Growth Rate:
42.17%
Total runs:
2.2K
Run Growth:
743
Growth Rate:
34.30%
Total runs:
2.0K
Run Growth:
210
Growth Rate:
10.38%
Total runs:
220
Run Growth:
-181
Growth Rate:
-82.27%
Total runs:
160
Run Growth:
44
Growth Rate:
26.04%
Total runs:
153
Run Growth:
-542
Growth Rate:
-354.25%
Total runs:
140
Run Growth:
-78
Growth Rate:
-55.71%
Total runs:
114
Run Growth:
22
Growth Rate:
19.30%
Total runs:
57
Run Growth:
35
Growth Rate:
61.40%
Total runs:
51
Run Growth:
-18
Growth Rate:
-35.29%
Total runs:
50
Run Growth:
29
Growth Rate:
60.42%
Total runs:
41
Run Growth:
-103
Growth Rate:
-264.10%
Total runs:
39
Run Growth:
12
Growth Rate:
30.77%
Total runs:
33
Run Growth:
-78
Growth Rate:
-243.75%
Total runs:
27
Run Growth:
9
Growth Rate:
40.91%
Total runs:
24
Run Growth:
3
Growth Rate:
12.50%
Total runs:
19
Run Growth:
3
Growth Rate:
15.79%
Total runs:
18
Run Growth:
9
Growth Rate:
50.00%
Total runs:
12
Run Growth:
2
Growth Rate:
16.67%
Total runs:
12
Run Growth:
2
Growth Rate:
16.67%
Total runs:
12
Run Growth:
2
Growth Rate:
16.67%
Total runs:
9
Run Growth:
2
Growth Rate:
22.22%
Total runs:
7
Run Growth:
-5
Growth Rate:
-71.43%
Total runs:
6
Run Growth:
-2
Growth Rate:
-33.33%
Total runs:
5
Run Growth:
-2
Growth Rate:
-40.00%
Total runs:
5
Run Growth:
-6
Growth Rate:
-120.00%
Total runs:
4
Run Growth:
4
Growth Rate:
100.00%
Total runs:
4
Run Growth:
4
Growth Rate:
100.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%