The model is optimized for evaluating factual consistency in
summarization
.
It is the main model from the paper (see "T5-11B w. ANLI + TrueTeacher full" in Table 1) which is based on a
T5-11B
(Raffel
et al., 2020)
fine-tuned with a mixture of the following datasets:
The TrueTeacher dataset contains model-generated summaries of articles from the train split of the
CNN/DailyMail
dataset
(Hermann et al., 2015)
which are annotated for factual consistency using
FLAN-PaLM 540B
(Chung et al.,2022)
.
Summaries were generated using summarization models which were trained on the
XSum
dataset
(Narayan et al., 2018)
.
The input format for the model is: "premise: GROUNDING_DOCUMENT hypothesis: HYPOTHESIS_SUMMARY".
To accomodate the input length of common summarization datasets we recommend setting
max_length
to
2048
.
The model predicts a binary label ('1' - Factualy Consistent, '0' - Factualy Inconsistent).
This model is intended for a research use (
non-commercial
) in English.
The recommended use case is evaluating factual consistency in summarization.
Out-of-scope use
Any use cases which violate the
cc-by-nc-4.0
license.
Usage in languages other than English.
Usage examples
classification
from transformers import T5ForConditionalGeneration
from transformers import T5Tokenizer
model_path = 'google/t5_11b_trueteacher_and_anli'
tokenizer = T5Tokenizer.from_pretrained(model_path)
model = T5ForConditionalGeneration.from_pretrained(model_path)
premise = 'the sun is shining'for hypothesis, expected in [('the sun is out in the sky', '1'),
('the cat is shiny', '0')]:
input_ids = tokenizer(
f'premise: {premise} hypothesis: {hypothesis}',
return_tensors='pt',
truncation=True,
max_length=2048).input_ids
outputs = model.generate(input_ids)
result = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(f'premise: {premise}')
print(f'hypothesis: {hypothesis}')
print(f'result: {result} (expected: {expected})\n')
scoring
from transformers import T5ForConditionalGeneration
from transformers import T5Tokenizer
import torch
model_path = 'google/t5_11b_trueteacher_and_anli'
tokenizer = T5Tokenizer.from_pretrained(model_path)
model = T5ForConditionalGeneration.from_pretrained(model_path)
premise = 'the sun is shining'for hypothesis, expected in [('the sun is out in the sky', '>> 0.5'),
('the cat is shiny', '<< 0.5')]:
input_ids = tokenizer(
f'premise: {premise} hypothesis: {hypothesis}',
return_tensors='pt',
truncation=True,
max_length=2048).input_ids
decoder_input_ids = torch.tensor([[tokenizer.pad_token_id]])
outputs = model(input_ids=input_ids, decoder_input_ids=decoder_input_ids)
logits = outputs.logits
probs = torch.softmax(logits[0], dim=-1)
one_token_id = tokenizer('1').input_ids[0]
entailment_prob = probs[0, one_token_id].item()
print(f'premise: {premise}')
print(f'hypothesis: {hypothesis}')
print(f'score: {entailment_prob:.3f} (expected: {expected})\n')
Citation
If you use this model for a research publication, please cite the TrueTeacher paper (using the bibtex entry below), as well as the ANLI, CNN/DailyMail, XSum, T5 and FLAN papers mentioned above.
@misc{gekhman2023trueteacher,
title={TrueTeacher: Learning Factual Consistency Evaluation with Large Language Models},
author={Zorik Gekhman and Jonathan Herzig and Roee Aharoni and Chen Elkind and Idan Szpektor},
year={2023},
eprint={2305.11171},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
Runs of google t5_11b_trueteacher_and_anli on huggingface.co
30.5K
Total runs
0
24-hour runs
-14
3-day runs
-2
7-day runs
30.0K
30-day runs
More Information About t5_11b_trueteacher_and_anli huggingface.co Model
More t5_11b_trueteacher_and_anli license Visit here:
t5_11b_trueteacher_and_anli huggingface.co is an AI model on huggingface.co that provides t5_11b_trueteacher_and_anli's model effect (), which can be used instantly with this google t5_11b_trueteacher_and_anli model. huggingface.co supports a free trial of the t5_11b_trueteacher_and_anli model, and also provides paid use of the t5_11b_trueteacher_and_anli. Support call t5_11b_trueteacher_and_anli model through api, including Node.js, Python, http.
t5_11b_trueteacher_and_anli huggingface.co is an online trial and call api platform, which integrates t5_11b_trueteacher_and_anli's modeling effects, including api services, and provides a free online trial of t5_11b_trueteacher_and_anli, you can try t5_11b_trueteacher_and_anli online for free by clicking the link below.
google t5_11b_trueteacher_and_anli online free url in huggingface.co:
t5_11b_trueteacher_and_anli is an open source model from GitHub that offers a free installation service, and any user can find t5_11b_trueteacher_and_anli on GitHub to install. At the same time, huggingface.co provides the effect of t5_11b_trueteacher_and_anli install, users can directly use t5_11b_trueteacher_and_anli installed effect in huggingface.co for debugging and trial. It also supports api for free installation.
t5_11b_trueteacher_and_anli install url in huggingface.co: