google / matcha-chartqa

huggingface.co
Total runs: 435
24-hour runs: 124
7-day runs: 66
30-day runs: -245
Model's Last Updated: July 22 2023
visual-question-answering

Introduction of matcha-chartqa

Model Details of matcha-chartqa

Model card for MatCha - fine-tuned on ChartQA

drawing

This model is the MatCha model, fine-tuned on Chart2text-pew dataset.

Table of Contents

  1. TL;DR
  2. Using the model
  3. Contribution
  4. Citation

TL;DR

The abstract of the paper states that:

Visual language data such as plots, charts, and infographics are ubiquitous in the human world. However, state-of-the-art visionlanguage models do not perform well on these data. We propose MATCHA (Math reasoning and Chart derendering pretraining) to enhance visual language models’ capabilities jointly modeling charts/plots and language data. Specifically we propose several pretraining tasks that cover plot deconstruction and numerical reasoning which are the key capabilities in visual language modeling. We perform the MATCHA pretraining starting from Pix2Struct, a recently proposed imageto-text visual language model. On standard benchmarks such as PlotQA and ChartQA, MATCHA model outperforms state-of-the-art methods by as much as nearly 20%. We also examine how well MATCHA pretraining transfers to domains such as screenshot, textbook diagrams, and document figures and observe overall improvement, verifying the usefulness of MATCHA pretraining on broader visual language tasks.

Using the model

You should ask specific questions to the model in order to get consistent generations. Here we are asking the model whether the sum of values that are in a chart are greater than the largest value.

from transformers import Pix2StructProcessor, Pix2StructForConditionalGeneration
import requests
from PIL import Image

processor = Pix2StructProcessor.from_pretrained('google/matcha-chartqa')
model = Pix2StructForConditionalGeneration.from_pretrained('google/matcha-chartqa')

url = "https://raw.githubusercontent.com/vis-nlp/ChartQA/main/ChartQA%20Dataset/val/png/20294671002019.png"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(images=image, text="Is the sum of all 4 places greater than Laos?", return_tensors="pt")
predictions = model.generate(**inputs, max_new_tokens=512)
print(processor.decode(predictions[0], skip_special_tokens=True))
>>> No

To run the predictions on GPU, simply add .to(0) when creating the model and when getting the inputs ( inputs = inputs.to(0) )

Converting from T5x to huggingface

You can use the convert_pix2struct_checkpoint_to_pytorch.py script as follows:

python convert_pix2struct_checkpoint_to_pytorch.py --t5x_checkpoint_path PATH_TO_T5X_CHECKPOINTS --pytorch_dump_path PATH_TO_SAVE --is_vqa

if you are converting a large model, run:

python convert_pix2struct_checkpoint_to_pytorch.py --t5x_checkpoint_path PATH_TO_T5X_CHECKPOINTS --pytorch_dump_path PATH_TO_SAVE --use-large --is_vqa

Once saved, you can push your converted model with the following snippet:

from transformers import Pix2StructForConditionalGeneration, Pix2StructProcessor

model = Pix2StructForConditionalGeneration.from_pretrained(PATH_TO_SAVE)
processor = Pix2StructProcessor.from_pretrained(PATH_TO_SAVE)

model.push_to_hub("USERNAME/MODEL_NAME")
processor.push_to_hub("USERNAME/MODEL_NAME")

Contribution

This model was originally contributed by Fangyu Liu, Francesco Piccinno et al. and added to the Hugging Face ecosystem by Younes Belkada .

Citation

If you want to cite this work, please consider citing the original paper:

@misc{liu2022matcha,
      title={MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering}, 
      author={Fangyu Liu and Francesco Piccinno and Syrine Krichene and Chenxi Pang and Kenton Lee and Mandar Joshi and Yasemin Altun and Nigel Collier and Julian Martin Eisenschlos},
      year={2022},
      eprint={2212.09662},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Runs of google matcha-chartqa on huggingface.co

435
Total runs
124
24-hour runs
26
3-day runs
66
7-day runs
-245
30-day runs

More Information About matcha-chartqa huggingface.co Model

More matcha-chartqa license Visit here:

https://choosealicense.com/licenses/apache-2.0

matcha-chartqa huggingface.co

matcha-chartqa huggingface.co is an AI model on huggingface.co that provides matcha-chartqa's model effect (), which can be used instantly with this google matcha-chartqa model. huggingface.co supports a free trial of the matcha-chartqa model, and also provides paid use of the matcha-chartqa. Support call matcha-chartqa model through api, including Node.js, Python, http.

matcha-chartqa huggingface.co Url

https://huggingface.co/google/matcha-chartqa

google matcha-chartqa online free

matcha-chartqa huggingface.co is an online trial and call api platform, which integrates matcha-chartqa's modeling effects, including api services, and provides a free online trial of matcha-chartqa, you can try matcha-chartqa online for free by clicking the link below.

google matcha-chartqa online free url in huggingface.co:

https://huggingface.co/google/matcha-chartqa

matcha-chartqa install

matcha-chartqa is an open source model from GitHub that offers a free installation service, and any user can find matcha-chartqa on GitHub to install. At the same time, huggingface.co provides the effect of matcha-chartqa install, users can directly use matcha-chartqa installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

matcha-chartqa install url in huggingface.co:

https://huggingface.co/google/matcha-chartqa

Url of matcha-chartqa

matcha-chartqa huggingface.co Url

Provider of matcha-chartqa huggingface.co

google
ORGANIZATIONS

Other API from google

huggingface.co

Total runs: 1.6M
Run Growth: 262.5K
Growth Rate: 16.23%
Updated: July 17 2023
huggingface.co

Total runs: 1.4M
Run Growth: -703.7K
Growth Rate: -51.68%
Updated: January 24 2023
huggingface.co

Total runs: 726.7K
Run Growth: -357.8K
Growth Rate: -50.94%
Updated: February 29 2024
huggingface.co

Total runs: 702.7K
Run Growth: -335.5K
Growth Rate: -47.30%
Updated: January 24 2023
huggingface.co

Total runs: 701.2K
Run Growth: -88.7K
Growth Rate: -12.82%
Updated: July 27 2023
huggingface.co

Total runs: 681.0K
Run Growth: -41.7K
Growth Rate: -6.26%
Updated: July 17 2023
huggingface.co

Total runs: 521.5K
Run Growth: -98.9K
Growth Rate: -19.14%
Updated: October 10 2023
huggingface.co

Total runs: 348.3K
Run Growth: -69.2K
Growth Rate: -19.92%
Updated: August 27 2024
huggingface.co

Total runs: 314.4K
Run Growth: -45.3K
Growth Rate: -14.86%
Updated: August 27 2024
huggingface.co

Total runs: 190.5K
Run Growth: 183.5K
Growth Rate: 99.72%
Updated: February 17 2023
huggingface.co

Total runs: 178.1K
Run Growth: -28.4K
Growth Rate: -15.95%
Updated: November 28 2023
huggingface.co

Total runs: 168.0K
Run Growth: -303.6K
Growth Rate: -173.20%
Updated: September 27 2024
huggingface.co

Total runs: 150.2K
Run Growth: -29.7K
Growth Rate: -20.28%
Updated: November 07 2023
huggingface.co

Total runs: 128.8K
Run Growth: -1.9K
Growth Rate: -1.49%
Updated: January 24 2023
huggingface.co

Total runs: 118.1K
Run Growth: -13.6K
Growth Rate: -11.30%
Updated: January 24 2023
huggingface.co

Total runs: 114.1K
Run Growth: -242.2K
Growth Rate: -211.00%
Updated: August 07 2024
huggingface.co

Total runs: 114.1K
Run Growth: -224.6K
Growth Rate: -195.69%
Updated: August 14 2024
huggingface.co

Total runs: 105.8K
Run Growth: -57.3K
Growth Rate: -54.21%
Updated: January 24 2023
huggingface.co

Total runs: 101.1K
Run Growth: -26.8K
Growth Rate: -27.39%
Updated: September 18 2023
huggingface.co

Total runs: 99.5K
Run Growth: 11.5K
Growth Rate: 11.63%
Updated: January 24 2023
huggingface.co

Total runs: 93.5K
Run Growth: -202.2K
Growth Rate: -204.95%
Updated: January 24 2023
huggingface.co

Total runs: 91.3K
Run Growth: -115.2K
Growth Rate: -131.43%
Updated: August 27 2024
huggingface.co

Total runs: 90.9K
Run Growth: -26.9K
Growth Rate: -29.35%
Updated: August 07 2024
huggingface.co

Total runs: 87.1K
Run Growth: 4.0K
Growth Rate: 4.68%
Updated: January 24 2023
huggingface.co

Total runs: 73.9K
Run Growth: -56.9K
Growth Rate: -77.22%
Updated: January 24 2023
huggingface.co

Total runs: 54.8K
Run Growth: -42.1K
Growth Rate: -77.39%
Updated: September 27 2024
huggingface.co

Total runs: 54.1K
Run Growth: -543.7K
Growth Rate: -1002.98%
Updated: August 07 2024
huggingface.co

Total runs: 51.2K
Run Growth: 8.0K
Growth Rate: 15.37%
Updated: January 24 2023
huggingface.co

Total runs: 37.8K
Run Growth: -25.7K
Growth Rate: -68.90%
Updated: June 27 2024
huggingface.co

Total runs: 36.6K
Run Growth: -150.4K
Growth Rate: -410.92%
Updated: April 29 2024
huggingface.co

Total runs: 13.9K
Run Growth: -12.7K
Growth Rate: -91.69%
Updated: January 24 2023
huggingface.co

Total runs: 8.9K
Run Growth: -1.9K
Growth Rate: -20.65%
Updated: September 06 2023
huggingface.co

Total runs: 8.9K
Run Growth: 44
Growth Rate: 0.50%
Updated: July 06 2023
huggingface.co

Total runs: 8.6K
Run Growth: 7.8K
Growth Rate: 90.88%
Updated: November 28 2023
huggingface.co

Total runs: 8.6K
Run Growth: -111.7K
Growth Rate: -1294.89%
Updated: September 06 2023
huggingface.co

Total runs: 8.0K
Run Growth: 5.7K
Growth Rate: 71.50%
Updated: August 07 2024
huggingface.co

Total runs: 7.9K
Run Growth: -3.1K
Growth Rate: -40.69%
Updated: April 29 2024
huggingface.co

Total runs: 6.9K
Run Growth: -5.4K
Growth Rate: -78.12%
Updated: November 27 2023
huggingface.co

Total runs: 6.4K
Run Growth: 406
Growth Rate: 6.37%
Updated: January 24 2023