Total runs:
2.7M
Run Growth:
560.0K
Growth Rate:
21.07%
stabilityai / japanese-instructblip-alpha
Model's Last Updated: November 17 2023
image-to-text
Introduction of japanese-instructblip-alpha
Model Details of japanese-instructblip-alpha
Japanese InstructBLIP Alpha
Model Details
Japanese InstructBLIP Alpha is a vision-language instruction-following model that enables to generate Japanese descriptions for input images and optionally input texts such as questions.
Usage
First install additional dependencies in requirements.txt :
pip install sentencepiece einops
import torch
from transformers import LlamaTokenizer, AutoModelForVision2Seq, BlipImageProcessor
from PIL import Image
import requests
# helper function to format input prompts
def build_prompt(prompt="", sep="\n\n### "):
sys_msg = "以下は、タスクを説明する指示と、文脈のある入力の組み合わせです。要求を適切に満たす応答を書きなさい。"
p = sys_msg
roles = ["指示", "応答"]
user_query = "与えられた画像について、詳細に述べてください。"
msgs = [": \n" + user_query, ": "]
if prompt:
roles.insert(1, "入力")
msgs.insert(1, ": \n" + prompt)
for role, msg in zip(roles, msgs):
p += sep + role + msg
return p
# load model
model = AutoModelForVision2Seq.from_pretrained("stabilityai/japanese-instructblip-alpha", trust_remote_code=True)
processor = BlipImageProcessor.from_pretrained("stabilityai/japanese-instructblip-alpha")
tokenizer = LlamaTokenizer.from_pretrained("novelai/nerdstash-tokenizer-v1", additional_special_tokens=['▁▁'])
device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)
# prepare inputs
url = "https://images.unsplash.com/photo-1582538885592-e70a5d7ab3d3?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=1770&q=80"
image = Image.open(requests.get(url, stream=True).raw).convert("RGB")
prompt = "" # input empty string for image captioning. You can also input questions as prompts
prompt = build_prompt(prompt)
inputs = processor(images=image, return_tensors="pt")
text_encoding = tokenizer(prompt, add_special_tokens=False, return_tensors="pt")
text_encoding["qformer_input_ids"] = text_encoding["input_ids"].clone()
text_encoding["qformer_attention_mask"] = text_encoding["attention_mask"].clone()
inputs.update(text_encoding)
# generate
outputs = model.generate(
**inputs.to(device, dtype=model.dtype),
num_beams=5,
max_new_tokens=32,
min_length=1,
)
generated_text = tokenizer.batch_decode(outputs, skip_special_tokens=True)[0].strip()
print(generated_text)
# 桜と東京スカイツリー
Model Details
- Developed by : Stability AI
- Model type : InstructBLIP
- Language(s) : Japanese
- License : JAPANESE STABLELM RESEARCH LICENSE AGREEMENT .
Training
Japanese InstructBLIP Alpha leverages the InstructBLIP architecture. It consists of 3 components: a frozen vision image encoder, a Q-Former, and a frozen LLM. The vision encoder and the Q-Former were initialized with Salesforce/instructblip-vicuna-7b . For the frozen LLM, Japanese-StableLM-Instruct-Alpha-7B model was used. During training, only Q-Former was trained.
Training Dataset
The training dataset includes the following public datasets:
- CC12M with captions translated into Japanese
- MS-COCO with STAIR Captions
- Japanese Visual Genome VQA dataset
Use and Limitations
Intended Use
This model is intended to be used by the open-source community in chat-like applications in adherence with the research license.
Limitations and bias
Although the aforementioned datasets help to steer the base language models into "safer" distributions of text, not all biases and toxicity can be mitigated through fine-tuning. We ask that users be mindful of such potential issues that can arise in generated responses. Do not treat model outputs as substitutes for human judgment or as sources of truth. Please use responsibly.
How to cite
@misc{JapaneseInstructBLIPAlpha,
url = {[https://huggingface.co/stabilityai/japanese-instructblip-alpha](https://huggingface.co/stabilityai/japanese-instructblip-alpha)},
title = {Japanese InstructBLIP Alpha},
author = {Shing, Makoto and Akiba, Takuya}
}
Citations
@misc{dai2023instructblip,
title = {InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning},
author = {Wenliang Dai and Junnan Li and Dongxu Li and Anthony Meng Huat Tiong and Junqi Zhao and Weisheng Wang and Boyang Li and Pascale Fung and Steven Hoi},
year = {2023},
eprint = {2305.06500},
archivePrefix = {arXiv},
primaryClass = {cs.CV}
}
Contact
- For questions and comments about the model, please join Stable Community Japan .
- For future announcements / information about Stability AI models, research, and events, please follow https://twitter.com/StabilityAI_JP .
- For business and partnership inquiries, please contact [email protected] . ビジネスや協業に関するお問い合わせは [email protected] にご連絡ください。
Runs of stabilityai japanese-instructblip-alpha on huggingface.co
390
Total runs
0
24-hour runs
13
3-day runs
44
7-day runs
230
30-day runs
More Information About japanese-instructblip-alpha huggingface.co Model
Url of japanese-instructblip-alpha
japanese-instructblip-alpha huggingface.co Url
Provider of japanese-instructblip-alpha huggingface.co
Other API from stabilityai
Updated: October 31 2023
Total runs:
1.1M
Run Growth:
423.3K
Growth Rate:
39.67%
Total runs:
1.1M
Run Growth:
114.7K
Growth Rate:
10.70%
Total runs:
852.6K
Run Growth:
613.1K
Growth Rate:
72.33%
Total runs:
425.6K
Run Growth:
88.3K
Growth Rate:
20.51%
Total runs:
388.8K
Run Growth:
224.1K
Growth Rate:
58.15%
Total runs:
277.7K
Run Growth:
-120.3K
Growth Rate:
-43.27%
Total runs:
249.3K
Run Growth:
68.4K
Growth Rate:
26.67%
Total runs:
244.7K
Run Growth:
-77.1K
Growth Rate:
-43.16%
Total runs:
168.2K
Run Growth:
33.6K
Growth Rate:
20.80%
Total runs:
135.0K
Run Growth:
3.4K
Growth Rate:
3.56%
Total runs:
134.3K
Run Growth:
64.6K
Growth Rate:
48.45%
Total runs:
123.9K
Run Growth:
-59.9K
Growth Rate:
-47.92%
Total runs:
90.6K
Run Growth:
-14.2K
Growth Rate:
-15.22%
Total runs:
66.1K
Run Growth:
19.8K
Growth Rate:
52.01%
Total runs:
45.8K
Run Growth:
-8.4K
Growth Rate:
-18.90%
Total runs:
41.6K
Run Growth:
37.7K
Growth Rate:
35.58%
Total runs:
35.0K
Run Growth:
3.2K
Growth Rate:
9.18%
Total runs:
28.2K
Run Growth:
-2.2K
Growth Rate:
-8.09%
Total runs:
27.3K
Run Growth:
5.1K
Growth Rate:
19.07%
Total runs:
24.7K
Run Growth:
2.8K
Growth Rate:
11.06%
Total runs:
19.6K
Run Growth:
-574
Growth Rate:
-2.97%
Total runs:
18.4K
Run Growth:
12.2K
Growth Rate:
66.31%
Total runs:
17.2K
Run Growth:
9.4K
Growth Rate:
54.20%
Total runs:
16.4K
Run Growth:
5.4K
Growth Rate:
32.47%
Total runs:
11.1K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
11.0K
Run Growth:
2.5K
Growth Rate:
22.73%
Total runs:
9.1K
Run Growth:
15
Growth Rate:
0.17%
Total runs:
8.4K
Run Growth:
542
Growth Rate:
6.49%
Total runs:
8.2K
Run Growth:
1.5K
Growth Rate:
17.79%
Total runs:
6.7K
Run Growth:
1.4K
Growth Rate:
21.33%
Total runs:
6.2K
Run Growth:
1.3K
Growth Rate:
21.52%
Total runs:
5.6K
Run Growth:
-884
Growth Rate:
-15.69%
Total runs:
5.6K
Run Growth:
1.1K
Growth Rate:
20.57%
Total runs:
4.6K
Run Growth:
-737
Growth Rate:
-15.50%
Total runs:
4.6K
Run Growth:
-1.2K
Growth Rate:
-25.95%
Total runs:
4.1K
Run Growth:
1.5K
Growth Rate:
39.53%
Total runs:
2.4K
Run Growth:
997
Growth Rate:
43.63%
Total runs:
2.3K
Run Growth:
664
Growth Rate:
29.66%
Total runs:
2.3K
Run Growth:
-582
Growth Rate:
-25.74%
Total runs:
2.1K
Run Growth:
-6.4K
Growth Rate:
-313.75%
Total runs:
1.9K
Run Growth:
470
Growth Rate:
26.11%
Total runs:
1.8K
Run Growth:
488
Growth Rate:
27.52%
Total runs:
1.6K
Run Growth:
-119
Growth Rate:
-7.57%
Total runs:
1.6K
Run Growth:
744
Growth Rate:
48.31%
Total runs:
1.5K
Run Growth:
676
Growth Rate:
46.27%
Total runs:
1.4K
Run Growth:
-484
Growth Rate:
-32.93%
Total runs:
1.4K
Run Growth:
336
Growth Rate:
25.40%
Total runs:
1.4K
Run Growth:
603
Growth Rate:
45.65%
Total runs:
1.4K
Run Growth:
482
Growth Rate:
36.46%
Total runs:
1.3K
Run Growth:
326
Growth Rate:
25.53%
Total runs:
1.1K
Run Growth:
574
Growth Rate:
49.65%
Total runs:
926
Run Growth:
257
Growth Rate:
29.27%
Total runs:
754
Run Growth:
0
Growth Rate:
0.00%
Total runs:
706
Run Growth:
474
Growth Rate:
67.14%
Total runs:
676
Run Growth:
575
Growth Rate:
85.06%
Total runs:
667
Run Growth:
92
Growth Rate:
100.00%
Total runs:
600
Run Growth:
0
Growth Rate:
0.00%
Total runs:
577
Run Growth:
41
Growth Rate:
7.17%
Total runs:
569
Run Growth:
0
Growth Rate:
0.00%
Total runs:
569
Run Growth:
197
Growth Rate:
31.07%
Total runs:
488
Run Growth:
271
Growth Rate:
100.00%
Total runs:
414
Run Growth:
410
Growth Rate:
92.76%
Total runs:
333
Run Growth:
10
Growth Rate:
3.00%
Total runs:
317
Run Growth:
-25
Growth Rate:
-8.01%
Total runs:
303
Run Growth:
342
Growth Rate:
93.44%
Total runs:
277
Run Growth:
-5.1K
Growth Rate:
-1846.93%
Total runs:
262
Run Growth:
-152
Growth Rate:
-57.36%
Total runs:
220
Run Growth:
-101
Growth Rate:
-47.42%
Total runs:
212
Run Growth:
-125
Growth Rate:
-59.24%
Total runs:
140
Run Growth:
28
Growth Rate:
20.00%
Total runs:
134
Run Growth:
-206
Growth Rate:
-160.94%
Total runs:
127
Run Growth:
274
Growth Rate:
100.00%
Total runs:
121
Run Growth:
79
Growth Rate:
70.54%
Total runs:
115
Run Growth:
-260
Growth Rate:
-224.14%
Total runs:
102
Run Growth:
37
Growth Rate:
35.92%
Total runs:
101
Run Growth:
48
Growth Rate:
100.00%
Total runs:
77
Run Growth:
25
Growth Rate:
100.00%
Total runs:
66
Run Growth:
-30
Growth Rate:
-45.45%
Total runs:
35
Run Growth:
-6
Growth Rate:
-16.22%
Total runs:
19
Run Growth:
-26
Growth Rate:
-136.84%
Total runs:
11
Run Growth:
-7
Growth Rate:
-63.64%
Total runs:
8
Run Growth:
0
Growth Rate:
0.00%
Total runs:
5
Run Growth:
2
Growth Rate:
40.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%