google / t5_11b_trueteacher_and_anli

huggingface.co
Total runs: 30.5K
24-hour runs: 0
7-day runs: -2
30-day runs: 30.0K
Model's Last Updated: December 26 2023
text2text-generation

Introduction of t5_11b_trueteacher_and_anli

Model Details of t5_11b_trueteacher_and_anli

TrueTeacher

This is a Factual Consistency Evaluation model, introduced in the TrueTeacher paper (Gekhman et al, 2023) .

Model Details

The model is optimized for evaluating factual consistency in summarization .

It is the main model from the paper (see "T5-11B w. ANLI + TrueTeacher full" in Table 1) which is based on a T5-11B (Raffel et al., 2020) fine-tuned with a mixture of the following datasets:

The TrueTeacher dataset contains model-generated summaries of articles from the train split of the CNN/DailyMail dataset (Hermann et al., 2015) which are annotated for factual consistency using FLAN-PaLM 540B (Chung et al.,2022) . Summaries were generated using summarization models which were trained on the XSum dataset (Narayan et al., 2018) .

The input format for the model is: "premise: GROUNDING_DOCUMENT hypothesis: HYPOTHESIS_SUMMARY". To accomodate the input length of common summarization datasets we recommend setting max_length to 2048 .

The model predicts a binary label ('1' - Factualy Consistent, '0' - Factualy Inconsistent).

Evaluation results

This model achieves the following ROC AUC results on the summarization subset of the TRUE benchmark (Honovich et al, 2022) :

MNBM QAGS-X FRANK SummEval QAGS-C Average
78.1 89.4 93.6 88.5 89.4 87.8
Intended Use

This model is intended for a research use ( non-commercial ) in English.

The recommended use case is evaluating factual consistency in summarization.

Out-of-scope use

Any use cases which violate the cc-by-nc-4.0 license.

Usage in languages other than English.

Usage examples
classification
from transformers import T5ForConditionalGeneration
from transformers import T5Tokenizer

model_path = 'google/t5_11b_trueteacher_and_anli'
tokenizer = T5Tokenizer.from_pretrained(model_path)
model = T5ForConditionalGeneration.from_pretrained(model_path)

premise = 'the sun is shining'
for hypothesis, expected in [('the sun is out in the sky', '1'), 
                             ('the cat is shiny', '0')]:
  input_ids = tokenizer(
      f'premise: {premise} hypothesis: {hypothesis}',
      return_tensors='pt',
      truncation=True,
      max_length=2048).input_ids
  outputs = model.generate(input_ids)
  result = tokenizer.decode(outputs[0], skip_special_tokens=True)
  print(f'premise: {premise}')
  print(f'hypothesis: {hypothesis}')
  print(f'result: {result} (expected: {expected})\n')
scoring
from transformers import T5ForConditionalGeneration
from transformers import T5Tokenizer
import torch

model_path = 'google/t5_11b_trueteacher_and_anli'
tokenizer = T5Tokenizer.from_pretrained(model_path)
model = T5ForConditionalGeneration.from_pretrained(model_path)

premise = 'the sun is shining'
for hypothesis, expected in [('the sun is out in the sky', '>> 0.5'), 
                             ('the cat is shiny', '<< 0.5')]:
  input_ids = tokenizer(
      f'premise: {premise} hypothesis: {hypothesis}',
      return_tensors='pt',
      truncation=True,
      max_length=2048).input_ids
  decoder_input_ids = torch.tensor([[tokenizer.pad_token_id]])
  outputs = model(input_ids=input_ids, decoder_input_ids=decoder_input_ids)
  logits = outputs.logits
  probs = torch.softmax(logits[0], dim=-1)
  one_token_id = tokenizer('1').input_ids[0]
  entailment_prob = probs[0, one_token_id].item()
  print(f'premise: {premise}')
  print(f'hypothesis: {hypothesis}')
  print(f'score: {entailment_prob:.3f} (expected: {expected})\n')
Citation

If you use this model for a research publication, please cite the TrueTeacher paper (using the bibtex entry below), as well as the ANLI, CNN/DailyMail, XSum, T5 and FLAN papers mentioned above.

@misc{gekhman2023trueteacher,
      title={TrueTeacher: Learning Factual Consistency Evaluation with Large Language Models}, 
      author={Zorik Gekhman and Jonathan Herzig and Roee Aharoni and Chen Elkind and Idan Szpektor},
      year={2023},
      eprint={2305.11171},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Runs of google t5_11b_trueteacher_and_anli on huggingface.co

30.5K
Total runs
0
24-hour runs
-14
3-day runs
-2
7-day runs
30.0K
30-day runs

More Information About t5_11b_trueteacher_and_anli huggingface.co Model

More t5_11b_trueteacher_and_anli license Visit here:

https://choosealicense.com/licenses/cc-by-nc-4.0

t5_11b_trueteacher_and_anli huggingface.co

t5_11b_trueteacher_and_anli huggingface.co is an AI model on huggingface.co that provides t5_11b_trueteacher_and_anli's model effect (), which can be used instantly with this google t5_11b_trueteacher_and_anli model. huggingface.co supports a free trial of the t5_11b_trueteacher_and_anli model, and also provides paid use of the t5_11b_trueteacher_and_anli. Support call t5_11b_trueteacher_and_anli model through api, including Node.js, Python, http.

t5_11b_trueteacher_and_anli huggingface.co Url

https://huggingface.co/google/t5_11b_trueteacher_and_anli

google t5_11b_trueteacher_and_anli online free

t5_11b_trueteacher_and_anli huggingface.co is an online trial and call api platform, which integrates t5_11b_trueteacher_and_anli's modeling effects, including api services, and provides a free online trial of t5_11b_trueteacher_and_anli, you can try t5_11b_trueteacher_and_anli online for free by clicking the link below.

google t5_11b_trueteacher_and_anli online free url in huggingface.co:

https://huggingface.co/google/t5_11b_trueteacher_and_anli

t5_11b_trueteacher_and_anli install

t5_11b_trueteacher_and_anli is an open source model from GitHub that offers a free installation service, and any user can find t5_11b_trueteacher_and_anli on GitHub to install. At the same time, huggingface.co provides the effect of t5_11b_trueteacher_and_anli install, users can directly use t5_11b_trueteacher_and_anli installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

t5_11b_trueteacher_and_anli install url in huggingface.co:

https://huggingface.co/google/t5_11b_trueteacher_and_anli

Url of t5_11b_trueteacher_and_anli

t5_11b_trueteacher_and_anli huggingface.co Url

Provider of t5_11b_trueteacher_and_anli huggingface.co

google
ORGANIZATIONS

Other API from google

huggingface.co

Total runs: 2.3M
Run Growth: 907.4K
Growth Rate: 38.20%
Updated: July 17 2023
huggingface.co

Total runs: 1.5M
Run Growth: 104.9K
Growth Rate: 7.06%
Updated: January 24 2023
huggingface.co

Total runs: 1.0M
Run Growth: 340.8K
Growth Rate: 34.63%
Updated: February 29 2024
huggingface.co

Total runs: 844.7K
Run Growth: 221.7K
Growth Rate: 26.72%
Updated: July 17 2023
huggingface.co

Total runs: 808.6K
Run Growth: 48.7K
Growth Rate: 6.02%
Updated: January 24 2023
huggingface.co

Total runs: 664.0K
Run Growth: 29.0K
Growth Rate: 4.32%
Updated: July 27 2023
huggingface.co

Total runs: 586.9K
Run Growth: 60.4K
Growth Rate: 10.45%
Updated: October 10 2023
huggingface.co

Total runs: 574.3K
Run Growth: 304.8K
Growth Rate: 53.08%
Updated: August 27 2024
huggingface.co

Total runs: 451.9K
Run Growth: 75.3K
Growth Rate: 16.66%
Updated: August 27 2024
huggingface.co

Total runs: 421.7K
Run Growth: 241.2K
Growth Rate: 57.91%
Updated: November 28 2023
huggingface.co

Total runs: 274.2K
Run Growth: 76.6K
Growth Rate: 27.92%
Updated: September 27 2024
huggingface.co

Total runs: 255.3K
Run Growth: 96.7K
Growth Rate: 38.05%
Updated: February 17 2023
huggingface.co

Total runs: 251.9K
Run Growth: 140.9K
Growth Rate: 55.92%
Updated: August 07 2024
huggingface.co

Total runs: 218.1K
Run Growth: 140.5K
Growth Rate: 64.43%
Updated: August 27 2024
huggingface.co

Total runs: 160.5K
Run Growth: 84.5K
Growth Rate: 52.61%
Updated: January 24 2023
huggingface.co

Total runs: 132.9K
Run Growth: 38.0K
Growth Rate: 28.58%
Updated: September 18 2023
huggingface.co

Total runs: 119.4K
Run Growth: 62.4K
Growth Rate: 52.25%
Updated: September 27 2024
huggingface.co

Total runs: 115.6K
Run Growth: 10.1K
Growth Rate: 8.74%
Updated: January 24 2023
huggingface.co

Total runs: 109.2K
Run Growth: 30.0K
Growth Rate: 27.48%
Updated: January 24 2023
huggingface.co

Total runs: 108.7K
Run Growth: 8.8K
Growth Rate: 8.10%
Updated: August 07 2024
huggingface.co

Total runs: 98.2K
Run Growth: -3.2K
Growth Rate: -3.26%
Updated: January 24 2023
huggingface.co

Total runs: 97.1K
Run Growth: -30.0K
Growth Rate: -30.75%
Updated: January 24 2023
huggingface.co

Total runs: 91.8K
Run Growth: 15.5K
Growth Rate: 17.24%
Updated: January 24 2023
huggingface.co

Total runs: 75.1K
Run Growth: -58.7K
Growth Rate: -71.57%
Updated: November 07 2023
huggingface.co

Total runs: 67.8K
Run Growth: 53.7K
Growth Rate: 79.76%
Updated: January 24 2023
huggingface.co

Total runs: 65.8K
Run Growth: 27.8K
Growth Rate: 42.26%
Updated: June 27 2024
huggingface.co

Total runs: 61.8K
Run Growth: -55.9K
Growth Rate: -90.46%
Updated: January 24 2023
huggingface.co

Total runs: 51.5K
Run Growth: -117.5K
Growth Rate: -228.09%
Updated: August 14 2024
huggingface.co

Total runs: 50.3K
Run Growth: 13.8K
Growth Rate: 28.02%
Updated: April 29 2024
huggingface.co

Total runs: 29.2K
Run Growth: -15.1K
Growth Rate: -47.60%
Updated: January 24 2023
huggingface.co

Total runs: 26.2K
Run Growth: 21.9K
Growth Rate: 84.62%
Updated: January 24 2023
huggingface.co

Total runs: 13.0K
Run Growth: 4.0K
Growth Rate: 28.71%
Updated: February 17 2023
huggingface.co

Total runs: 12.4K
Run Growth: 8.5K
Growth Rate: 68.17%
Updated: January 24 2023
huggingface.co

Total runs: 11.8K
Run Growth: 5.3K
Growth Rate: 43.84%
Updated: April 29 2024
huggingface.co

Total runs: 11.3K
Run Growth: -43.6K
Growth Rate: -385.61%
Updated: August 07 2024
huggingface.co

Total runs: 10.7K
Run Growth: 10.3K
Growth Rate: 96.28%
Updated: July 22 2023
huggingface.co

Total runs: 10.0K
Run Growth: 3.3K
Growth Rate: 33.99%
Updated: November 27 2023