Total runs:
1.8M
Run Growth:
259.7K
Growth Rate:
14.50%
Salesforce / blip-image-captioning-large
Model's Last Updated: December 07 2023
image-to-text
Introduction of blip-image-captioning-large
Model Details of blip-image-captioning-large
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
Model card for image captioning pretrained on COCO dataset - base architecture (with ViT large backbone).
|
|---|
| Pull figure from BLIP official repo |
TL;DR
Authors from the paper write in the abstract:
Vision-Language Pre-training (VLP) has advanced the performance for many vision-language tasks. However, most existing pre-trained models only excel in either understanding-based tasks or generation-based tasks. Furthermore, performance improvement has been largely achieved by scaling up the dataset with noisy image-text pairs collected from the web, which is a suboptimal source of supervision. In this paper, we propose BLIP, a new VLP framework which transfers flexibly to both vision-language understanding and generation tasks. BLIP effectively utilizes the noisy web data by bootstrapping the captions, where a captioner generates synthetic captions and a filter removes the noisy ones. We achieve state-of-the-art results on a wide range of vision-language tasks, such as image-text retrieval (+2.7% in average recall@1), image captioning (+2.8% in CIDEr), and VQA (+1.6% in VQA score). BLIP also demonstrates strong generalization ability when directly transferred to videolanguage tasks in a zero-shot manner. Code, models, and datasets are released.
Usage
You can use this model for conditional and un-conditional image captioning
Using the Pytorch model
Running the model on CPU
Click to expand
import requests
from PIL import Image
from transformers import BlipProcessor, BlipForConditionalGeneration
processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-large")
model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-large")
img_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg'
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')
# conditional image captioning
text = "a photography of"
inputs = processor(raw_image, text, return_tensors="pt")
out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))
# unconditional image captioning
inputs = processor(raw_image, return_tensors="pt")
out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))
Running the model on GPU
In full precision
Click to expand
import requests
from PIL import Image
from transformers import BlipProcessor, BlipForConditionalGeneration
processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-large")
model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-large").to("cuda")
img_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg'
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')
# conditional image captioning
text = "a photography of"
inputs = processor(raw_image, text, return_tensors="pt").to("cuda")
out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))
# unconditional image captioning
inputs = processor(raw_image, return_tensors="pt").to("cuda")
out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))
In half precision (
float16
)
Click to expand
import torch
import requests
from PIL import Image
from transformers import BlipProcessor, BlipForConditionalGeneration
processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-large")
model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-large", torch_dtype=torch.float16).to("cuda")
img_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg'
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')
# conditional image captioning
text = "a photography of"
inputs = processor(raw_image, text, return_tensors="pt").to("cuda", torch.float16)
out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))
# >>> a photography of a woman and her dog
# unconditional image captioning
inputs = processor(raw_image, return_tensors="pt").to("cuda", torch.float16)
out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))
>>> a woman sitting on the beach with her dog
BibTex and citation info
@misc{https://doi.org/10.48550/arxiv.2201.12086,
doi = {10.48550/ARXIV.2201.12086},
url = {https://arxiv.org/abs/2201.12086},
author = {Li, Junnan and Li, Dongxu and Xiong, Caiming and Hoi, Steven},
keywords = {Computer Vision and Pattern Recognition (cs.CV), FOS: Computer and information sciences, FOS: Computer and information sciences},
title = {BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation},
publisher = {arXiv},
year = {2022},
copyright = {Creative Commons Attribution 4.0 International}
}
Runs of Salesforce blip-image-captioning-large on huggingface.co
2.2M
Total runs
20.8K
24-hour runs
142.1K
3-day runs
-334.8K
7-day runs
134.7K
30-day runs
More Information About blip-image-captioning-large huggingface.co Model
Url of blip-image-captioning-large
blip-image-captioning-large huggingface.co Url
Provider of blip-image-captioning-large huggingface.co
Other API from Salesforce
Updated: August 01 2023
Total runs:
491.5K
Run Growth:
197.6K
Growth Rate:
40.20%
Total runs:
285.9K
Run Growth:
283.0K
Growth Rate:
98.99%
Total runs:
272.2K
Run Growth:
61.4K
Growth Rate:
22.59%
Total runs:
254.5K
Run Growth:
36.2K
Growth Rate:
14.55%
Total runs:
238.2K
Run Growth:
63.6K
Growth Rate:
26.69%
Total runs:
125.4K
Run Growth:
73.5K
Growth Rate:
58.57%
Total runs:
120.1K
Run Growth:
-257.6K
Growth Rate:
-214.44%
Total runs:
111.4K
Run Growth:
34.0K
Growth Rate:
29.34%
Total runs:
110.3K
Run Growth:
-276.3K
Growth Rate:
-250.47%
Total runs:
63.7K
Run Growth:
54.3K
Growth Rate:
85.33%
Total runs:
41.9K
Run Growth:
-19.5K
Growth Rate:
-42.53%
Total runs:
42.2K
Run Growth:
23.3K
Growth Rate:
55.30%
Total runs:
36.1K
Run Growth:
24.5K
Growth Rate:
67.91%
Total runs:
35.8K
Run Growth:
-56.7K
Growth Rate:
-158.24%
Total runs:
33.4K
Run Growth:
-22.6K
Growth Rate:
-67.89%
Total runs:
27.9K
Run Growth:
12.2K
Growth Rate:
43.59%
Total runs:
20.8K
Run Growth:
4.8K
Growth Rate:
22.30%
Total runs:
18.3K
Run Growth:
6.5K
Growth Rate:
35.56%
Total runs:
17.3K
Run Growth:
5.4K
Growth Rate:
31.06%
Total runs:
16.3K
Run Growth:
2.6K
Growth Rate:
15.65%
Total runs:
13.6K
Run Growth:
7.9K
Growth Rate:
58.12%
Total runs:
12.8K
Run Growth:
-25.2K
Growth Rate:
-197.55%
Total runs:
11.9K
Run Growth:
10.8K
Growth Rate:
90.69%
Total runs:
9.2K
Run Growth:
3.3K
Growth Rate:
35.54%
Total runs:
8.6K
Run Growth:
501
Growth Rate:
5.83%
Total runs:
5.5K
Run Growth:
5.1K
Growth Rate:
93.03%
Total runs:
4.7K
Run Growth:
3.3K
Growth Rate:
68.83%
Total runs:
4.7K
Run Growth:
3.7K
Growth Rate:
79.53%
Total runs:
3.8K
Run Growth:
663
Growth Rate:
17.42%
Total runs:
3.3K
Run Growth:
-1.4K
Growth Rate:
-43.38%
Total runs:
3.3K
Run Growth:
-4.4K
Growth Rate:
-132.45%
Total runs:
2.5K
Run Growth:
-7.8K
Growth Rate:
-304.36%
Total runs:
1.8K
Run Growth:
-1.6K
Growth Rate:
-83.34%
Total runs:
1.8K
Run Growth:
684
Growth Rate:
37.15%
Total runs:
1.5K
Run Growth:
1.1K
Growth Rate:
74.62%
Total runs:
1.4K
Run Growth:
11
Growth Rate:
0.81%
Total runs:
1.3K
Run Growth:
-6.9K
Growth Rate:
-535.02%
Total runs:
1.1K
Run Growth:
-152
Growth Rate:
-14.02%
Total runs:
889
Run Growth:
353
Growth Rate:
39.71%
Total runs:
869
Run Growth:
-1.2K
Growth Rate:
-139.13%
Total runs:
865
Run Growth:
-1.4K
Growth Rate:
-166.13%
Total runs:
850
Run Growth:
-1.1K
Growth Rate:
-131.41%
Total runs:
793
Run Growth:
-19
Growth Rate:
-2.40%
Total runs:
769
Run Growth:
49
Growth Rate:
6.34%
Total runs:
727
Run Growth:
223
Growth Rate:
30.67%
Total runs:
689
Run Growth:
-913
Growth Rate:
-132.51%
Total runs:
583
Run Growth:
386
Growth Rate:
66.21%
Total runs:
580
Run Growth:
-3.8K
Growth Rate:
-659.14%
Total runs:
476
Run Growth:
-2.2K
Growth Rate:
-468.91%
Total runs:
446
Run Growth:
72
Growth Rate:
16.14%
Total runs:
385
Run Growth:
139
Growth Rate:
36.10%
Total runs:
351
Run Growth:
-592
Growth Rate:
-168.66%
Total runs:
245
Run Growth:
-190
Growth Rate:
-77.55%
Total runs:
178
Run Growth:
-207
Growth Rate:
-118.97%
Total runs:
128
Run Growth:
-122
Growth Rate:
-95.31%
Total runs:
98
Run Growth:
91
Growth Rate:
92.86%
Total runs:
71
Run Growth:
-100
Growth Rate:
-140.85%
Total runs:
66
Run Growth:
8
Growth Rate:
12.12%
Total runs:
53
Run Growth:
-126
Growth Rate:
-196.88%
Total runs:
47
Run Growth:
-59
Growth Rate:
-137.21%
Total runs:
41
Run Growth:
13
Growth Rate:
30.23%
Total runs:
42
Run Growth:
-29
Growth Rate:
-69.05%
Total runs:
27
Run Growth:
8
Growth Rate:
29.63%
Total runs:
18
Run Growth:
-4
Growth Rate:
-23.53%
Total runs:
17
Run Growth:
-67
Growth Rate:
-418.75%
Total runs:
13
Run Growth:
-5
Growth Rate:
-38.46%
Total runs:
11
Run Growth:
1
Growth Rate:
9.09%
Total runs:
10
Run Growth:
5
Growth Rate:
50.00%
Total runs:
10
Run Growth:
-23
Growth Rate:
-230.00%
Total runs:
7
Run Growth:
3
Growth Rate:
42.86%
Total runs:
6
Run Growth:
-3
Growth Rate:
-50.00%
Total runs:
6
Run Growth:
-46
Growth Rate:
-766.67%
Total runs:
5
Run Growth:
-21
Growth Rate:
-420.00%
Total runs:
6
Run Growth:
-4
Growth Rate:
-80.00%
Total runs:
6
Run Growth:
-1
Growth Rate:
-20.00%
Total runs:
3
Run Growth:
-14
Growth Rate:
-466.67%
Total runs:
4
Run Growth:
-1
Growth Rate:
-33.33%
Total runs:
3
Run Growth:
3
Growth Rate:
100.00%
Total runs:
3
Run Growth:
-15
Growth Rate:
-500.00%
Total runs:
3
Run Growth:
-3
Growth Rate:
-100.00%
Total runs:
4
Run Growth:
-2
Growth Rate:
-66.67%
Total runs:
2
Run Growth:
-1
Growth Rate:
-50.00%
Total runs:
2
Run Growth:
2
Growth Rate:
100.00%
Total runs:
2
Run Growth:
0
Growth Rate:
0.00%
Total runs:
2
Run Growth:
-1
Growth Rate:
-50.00%