ai-forever / FRED-T5-large

huggingface.co
Total runs: 845
24-hour runs: 1
7-day runs: 20
30-day runs: 466
Model's Last Updated: Dezember 05 2023
text2text-generation

Introduction of FRED-T5-large

Model Details of FRED-T5-large

FRED-T5 large 820M (Full-scale Russian Enhanced Denoisers T5)

The model architecture design, pretraining, and evaluation are documented in our preprint: A Family of Pretrained Transformer Language Models for Russian .

The model was trained by SberDevices .

Architecture based on T5.

It has 24 layers and 1024 hidden size. More details in config.json.

The model trained on a mixture of 7 denoisers like UL2 with several differences ( https://arxiv.org/abs/2205.05131 ).

It was trained on Russian language corpus (300GB). The dataset is the same as for ruT5 models.

Bbpe tokenizer. 50257 + special tokens 107. Prefix tokens: '<LM>', '<SC1>',.. '<SC6>'

First half of the time model trained on the small part of all dataset (1%,3GB) and without prefixes in each task.

For RSG, we trained as described in the T5 paper. First, we trained multitask for all tasks. Then we took the best checkpoint for the task and trained it further. RSG submit here https://russiansuperglue.com/login/submit_info/2060

Total training time was around 35 days on 160 V100 GPUs + 5 days on 80 A100.

Usage (HuggingFace Models Repository)
import torch
from transformers import GPT2Tokenizer, T5ForConditionalGeneration 
tokenizer = GPT2Tokenizer.from_pretrained('ai-forever/FRED-T5-1.7B',eos_token='</s>')
model = T5ForConditionalGeneration.from_pretrained('ai-forever/FRED-T5-1.7B')
device='cuda'
model.to(device)

#Prefix <LM>
lm_text='<LM>Принялся Кутузов рассказывать свою историю как он сюда попал. Началось'
input_ids=torch.tensor([tokenizer.encode(lm_text)]).to(device)
outputs=model.generate(input_ids,eos_token_id=tokenizer.eos_token_id,early_stopping=True)
print(tokenizer.decode(outputs[0][1:]))

# print result: , как водится, с того, что он был в плену.</s>

#Prefix <SC1>
lm_text='<SC1>Принялся Кутузов рассказывать свою историю <extra_id_0>. Началось с того, что он был в армии, служил в артиллерии.'
input_ids=torch.tensor([tokenizer.encode(lm_text)]).to(device)
outputs=model.generate(input_ids,eos_token_id=tokenizer.eos_token_id,early_stopping=True)
print(tokenizer.decode(outputs[0][1:]))

#print result: '<extra_id_0>, как он жил</s>'

# Prefix <SC5>
lm_text='<SC5>Принялся Кутузов рассказывать свою историю <extra_id_0>. Началось с того, что он был в армии, служил в артиллерии.'
input_ids=torch.tensor([tokenizer.encode(lm_text)]).to(device)
outputs=model.generate(input_ids,eos_token_id=tokenizer.eos_token_id,early_stopping=True,max_length=100)
print(tokenizer.decode(outputs[0][1:]))

#print result: '<extra_id_0> </s>'

Authors

  • NLP core team RnD Telegram channel :
    • Dmitry Zmitrovich
    • Andrei Kalmykov
    • Vitaly Kadulin
    • Mikhail Novikov
    • Alexey Khoroshilov

Salute AI Community .

Cite us

@misc{zmitrovich2023family,
      title={A Family of Pretrained Transformer Language Models for Russian}, 
      author={Dmitry Zmitrovich and Alexander Abramov and Andrey Kalmykov and Maria Tikhonova and Ekaterina Taktasheva and Danil Astafurov and Mark Baushenko and Artem Snegirev and Tatiana Shavrina and Sergey Markov and Vladislav Mikhailov and Alena Fenogenova},
      year={2023},
      eprint={2309.10931},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Runs of ai-forever FRED-T5-large on huggingface.co

845
Total runs
1
24-hour runs
7
3-day runs
20
7-day runs
466
30-day runs

More Information About FRED-T5-large huggingface.co Model

More FRED-T5-large license Visit here:

https://choosealicense.com/licenses/apache-2.0

FRED-T5-large huggingface.co

FRED-T5-large huggingface.co is an AI model on huggingface.co that provides FRED-T5-large's model effect (), which can be used instantly with this ai-forever FRED-T5-large model. huggingface.co supports a free trial of the FRED-T5-large model, and also provides paid use of the FRED-T5-large. Support call FRED-T5-large model through api, including Node.js, Python, http.

ai-forever FRED-T5-large online free

FRED-T5-large huggingface.co is an online trial and call api platform, which integrates FRED-T5-large's modeling effects, including api services, and provides a free online trial of FRED-T5-large, you can try FRED-T5-large online for free by clicking the link below.

ai-forever FRED-T5-large online free url in huggingface.co:

https://huggingface.co/ai-forever/FRED-T5-large

FRED-T5-large install

FRED-T5-large is an open source model from GitHub that offers a free installation service, and any user can find FRED-T5-large on GitHub to install. At the same time, huggingface.co provides the effect of FRED-T5-large install, users can directly use FRED-T5-large installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

FRED-T5-large install url in huggingface.co:

https://huggingface.co/ai-forever/FRED-T5-large

Url of FRED-T5-large

FRED-T5-large huggingface.co Url

Provider of FRED-T5-large huggingface.co

ai-forever
ORGANIZATIONS

Other API from ai-forever

huggingface.co

Total runs: 525.5K
Run Growth: 485.9K
Growth Rate: 96.59%
Updated: November 03 2023
huggingface.co

Total runs: 10.6K
Run Growth: 1.6K
Growth Rate: 14.99%
Updated: Dezember 05 2023
huggingface.co

Total runs: 8.2K
Run Growth: 5.1K
Growth Rate: 59.45%
Updated: Dezember 29 2024
huggingface.co

Total runs: 5.9K
Run Growth: 3.6K
Growth Rate: 61.73%
Updated: Dezember 11 2023
huggingface.co

Total runs: 2.3K
Run Growth: -360
Growth Rate: -15.50%
Updated: Dezember 05 2023
huggingface.co

Total runs: 1.8K
Run Growth: 78
Growth Rate: 4.35%
Updated: Dezember 28 2023
huggingface.co

Total runs: 315
Run Growth: 165
Growth Rate: 52.05%
Updated: Januar 26 2023
huggingface.co

Total runs: 0
Run Growth: 0
Growth Rate: 0.00%
Updated: Dezember 24 2021
huggingface.co

Total runs: 0
Run Growth: 0
Growth Rate: 0.00%
Updated: Juni 08 2023
huggingface.co

Total runs: 0
Run Growth: 0
Growth Rate: 0.00%
Updated: September 21 2021