Total runs:
1.9M
Run Growth:
597.1K
Growth Rate:
32.42%
Salesforce / blip-itm-base-coco
Model's Last Updated: February 03 2025
Introduction of blip-itm-base-coco
Model Details of blip-itm-base-coco
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
Model card for BLIP trained on image-text matching - base architecture (with ViT base backbone) trained on COCO dataset.
|
|---|
| Pull figure from BLIP official repo |
TL;DR
Authors from the paper write in the abstract:
Vision-Language Pre-training (VLP) has advanced the performance for many vision-language tasks. However, most existing pre-trained models only excel in either understanding-based tasks or generation-based tasks. Furthermore, performance improvement has been largely achieved by scaling up the dataset with noisy image-text pairs collected from the web, which is a suboptimal source of supervision. In this paper, we propose BLIP, a new VLP framework which transfers flexibly to both vision-language understanding and generation tasks. BLIP effectively utilizes the noisy web data by bootstrapping the captions, where a captioner generates synthetic captions and a filter removes the noisy ones. We achieve state-of-the-art results on a wide range of vision-language tasks, such as image-text retrieval (+2.7% in average recall@1), image captioning (+2.8% in CIDEr), and VQA (+1.6% in VQA score). BLIP also demonstrates strong generalization ability when directly transferred to videolanguage tasks in a zero-shot manner. Code, models, and datasets are released.
Usage
You can use this model for conditional and un-conditional image captioning
Using the Pytorch model
Running the model on CPU
Click to expand
import requests
from PIL import Image
from transformers import BlipProcessor, BlipForImageTextRetrieval
processor = BlipProcessor.from_pretrained("Salesforce/blip-itm-base-coco")
model = BlipForImageTextRetrieval.from_pretrained("Salesforce/blip-itm-base-coco")
img_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg'
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')
question = "A woman and a dog sitting together in a beach."
inputs = processor(raw_image, question, return_tensors="pt")
itm_scores = model(**inputs)[0]
cosine_score = model(**inputs, use_itm_head=False)[0]
Running the model on GPU
In full precision
Click to expand
import requests
from PIL import Image
from transformers import BlipProcessor, BlipForImageTextRetrieval
processor = BlipProcessor.from_pretrained("Salesforce/blip-itm-base-coco")
model = BlipForImageTextRetrieval.from_pretrained("Salesforce/blip-itm-base-coco").to("cuda")
img_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg'
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')
question = "A woman and a dog sitting together in a beach."
inputs = processor(raw_image, question, return_tensors="pt").to("cuda")
itm_scores = model(**inputs)[0]
cosine_score = model(**inputs, use_itm_head=False)[0]
In half precision (
float16
)
Click to expand
import torch
import requests
from PIL import Image
from transformers import BlipProcessor, BlipForImageTextRetrieval
processor = BlipProcessor.from_pretrained("Salesforce/blip-itm-base-coco")
model = BlipForImageTextRetrieval.from_pretrained("Salesforce/blip-itm-base-coco", torch_dtype=torch.float16).to("cuda")
img_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg'
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')
question = "A woman and a dog sitting together in a beach."
inputs = processor(raw_image, question, return_tensors="pt").to("cuda", torch.float16)
itm_scores = model(**inputs)[0]
cosine_score = model(**inputs, use_itm_head=False)[0]
BibTex and citation info
@misc{https://doi.org/10.48550/arxiv.2201.12086,
doi = {10.48550/ARXIV.2201.12086},
url = {https://arxiv.org/abs/2201.12086},
author = {Li, Junnan and Li, Dongxu and Xiong, Caiming and Hoi, Steven},
keywords = {Computer Vision and Pattern Recognition (cs.CV), FOS: Computer and information sciences, FOS: Computer and information sciences},
title = {BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation},
publisher = {arXiv},
year = {2022},
copyright = {Creative Commons Attribution 4.0 International}
}
Runs of Salesforce blip-itm-base-coco on huggingface.co
24.5K
Total runs
0
24-hour runs
-88
3-day runs
1.9K
7-day runs
-1.7K
30-day runs
More Information About blip-itm-base-coco huggingface.co Model
Url of blip-itm-base-coco
blip-itm-base-coco huggingface.co Url
Provider of blip-itm-base-coco huggingface.co
Other API from Salesforce
Updated: February 03 2025
Total runs:
999.1K
Run Growth:
-331.8K
Growth Rate:
-32.59%
Total runs:
874.0K
Run Growth:
865.9K
Growth Rate:
99.05%
Total runs:
319.8K
Run Growth:
40.0K
Growth Rate:
12.61%
Total runs:
312.2K
Run Growth:
66.0K
Growth Rate:
22.89%
Total runs:
253.9K
Run Growth:
16.1K
Growth Rate:
6.34%
Total runs:
240.8K
Run Growth:
37.5K
Growth Rate:
93.70%
Total runs:
184.1K
Run Growth:
-96.9K
Growth Rate:
-54.73%
Total runs:
183.4K
Run Growth:
60.1K
Growth Rate:
32.84%
Total runs:
110.9K
Run Growth:
40.7K
Growth Rate:
37.34%
Total runs:
83.6K
Run Growth:
47.0K
Growth Rate:
56.58%
Total runs:
59.8K
Run Growth:
6.6K
Growth Rate:
11.12%
Total runs:
53.6K
Run Growth:
-3.8K
Growth Rate:
-7.17%
Total runs:
51.1K
Run Growth:
23.2K
Growth Rate:
45.05%
Total runs:
39.7K
Run Growth:
11.9K
Growth Rate:
88.03%
Total runs:
33.8K
Run Growth:
1.5K
Growth Rate:
4.10%
Total runs:
32.2K
Run Growth:
13.8K
Growth Rate:
43.04%
Total runs:
27.5K
Run Growth:
8.6K
Growth Rate:
30.77%
Total runs:
24.7K
Run Growth:
-8.6K
Growth Rate:
-28.35%
Total runs:
24.5K
Run Growth:
11.8K
Growth Rate:
48.02%
Total runs:
20.5K
Run Growth:
6.6K
Growth Rate:
32.41%
Total runs:
17.4K
Run Growth:
-33.2K
Growth Rate:
-197.80%
Total runs:
15.7K
Run Growth:
-19.6K
Growth Rate:
-126.06%
Total runs:
10.5K
Run Growth:
2.6K
Growth Rate:
34.07%
Total runs:
10.3K
Run Growth:
2.0K
Growth Rate:
21.77%
Total runs:
8.7K
Run Growth:
140
Growth Rate:
1.63%
Total runs:
8.7K
Run Growth:
2.9K
Growth Rate:
32.86%
Total runs:
8.1K
Run Growth:
3.7K
Growth Rate:
46.80%
Total runs:
6.4K
Run Growth:
-5.3K
Growth Rate:
-85.95%
Total runs:
5.3K
Run Growth:
1.4K
Growth Rate:
25.88%
Total runs:
5.2K
Run Growth:
-1.7K
Growth Rate:
-32.45%
Total runs:
3.9K
Run Growth:
1.2K
Growth Rate:
30.03%
Total runs:
2.9K
Run Growth:
-38
Growth Rate:
-1.38%
Total runs:
2.9K
Run Growth:
2.4K
Growth Rate:
82.85%
Total runs:
2.7K
Run Growth:
1.5K
Growth Rate:
53.63%
Total runs:
2.6K
Run Growth:
-12
Growth Rate:
-0.47%
Total runs:
2.5K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
2.5K
Run Growth:
2.1K
Growth Rate:
88.40%
Total runs:
2.4K
Run Growth:
847
Growth Rate:
34.96%
Total runs:
2.4K
Run Growth:
1.5K
Growth Rate:
62.40%
Total runs:
2.2K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1.9K
Run Growth:
1.5K
Growth Rate:
82.23%
Total runs:
1.9K
Run Growth:
673
Growth Rate:
37.10%
Total runs:
1.7K
Run Growth:
-3.2K
Growth Rate:
-185.86%
Total runs:
1.7K
Run Growth:
504
Growth Rate:
30.88%
Total runs:
1.6K
Run Growth:
-3.1K
Growth Rate:
-206.04%
Total runs:
1.5K
Run Growth:
1.2K
Growth Rate:
75.48%
Total runs:
1.5K
Run Growth:
1.5K
Growth Rate:
97.23%
Total runs:
1.5K
Run Growth:
869
Growth Rate:
59.24%
Total runs:
1.5K
Run Growth:
533
Growth Rate:
37.91%
Total runs:
1.4K
Run Growth:
705
Growth Rate:
52.65%
Total runs:
1.3K
Run Growth:
-72
Growth Rate:
-5.71%
Total runs:
1.2K
Run Growth:
149
Growth Rate:
12.20%
Total runs:
1.1K
Run Growth:
110
Growth Rate:
10.83%
Total runs:
972
Run Growth:
430
Growth Rate:
48.48%
Total runs:
911
Run Growth:
89
Growth Rate:
9.86%
Total runs:
809
Run Growth:
223
Growth Rate:
27.84%
Total runs:
794
Run Growth:
414
Growth Rate:
53.42%
Total runs:
776
Run Growth:
131
Growth Rate:
16.69%
Total runs:
741
Run Growth:
-2.0K
Growth Rate:
-246.20%
Total runs:
729
Run Growth:
506
Growth Rate:
73.23%
Total runs:
623
Run Growth:
-19.0K
Growth Rate:
-3083.58%
Total runs:
596
Run Growth:
0
Growth Rate:
0.00%
Total runs:
475
Run Growth:
104
Growth Rate:
21.99%
Total runs:
457
Run Growth:
75
Growth Rate:
16.45%
Total runs:
447
Run Growth:
192
Growth Rate:
45.93%
Total runs:
266
Run Growth:
-260
Growth Rate:
-100.00%
Total runs:
249
Run Growth:
44
Growth Rate:
18.33%
Total runs:
240
Run Growth:
119
Growth Rate:
58.05%
Total runs:
208
Run Growth:
-25
Growth Rate:
-12.56%
Total runs:
201
Run Growth:
183
Growth Rate:
86.73%
Total runs:
197
Run Growth:
-165
Growth Rate:
-85.05%
Total runs:
186
Run Growth:
-8.8K
Growth Rate:
-4546.63%
Total runs:
175
Run Growth:
-199
Growth Rate:
-114.37%
Total runs:
168
Run Growth:
161
Growth Rate:
95.83%
Total runs:
165
Run Growth:
159
Growth Rate:
97.55%
Total runs:
165
Run Growth:
145
Growth Rate:
86.83%
Total runs:
165
Run Growth:
156
Growth Rate:
94.55%
Total runs:
164
Run Growth:
155
Growth Rate:
96.27%
Total runs:
163
Run Growth:
145
Growth Rate:
88.96%
Total runs:
162
Run Growth:
152
Growth Rate:
93.83%
Total runs:
161
Run Growth:
149
Growth Rate:
92.55%
Total runs:
159
Run Growth:
149
Growth Rate:
93.71%
Total runs:
159
Run Growth:
150
Growth Rate:
95.54%
Total runs:
158
Run Growth:
150
Growth Rate:
94.94%
Total runs:
157
Run Growth:
150
Growth Rate:
95.54%
Total runs:
156
Run Growth:
-519
Growth Rate:
-328.48%
Total runs:
146
Run Growth:
-206
Growth Rate:
-141.10%
Total runs:
137
Run Growth:
-349
Growth Rate:
-262.41%
Total runs:
135
Run Growth:
-83
Growth Rate:
-65.35%
Total runs:
128
Run Growth:
-96
Growth Rate:
-121.52%
Total runs:
121
Run Growth:
92
Growth Rate:
74.80%
Total runs:
117
Run Growth:
108
Growth Rate:
92.31%
Total runs:
115
Run Growth:
0
Growth Rate:
0.00%