ai-forever / FRED-T5-1.7B

huggingface.co
Total runs: 2.2K
24-hour runs: -10
7-day runs: 351
30-day runs: 716
Model's Last Updated: 12月 05 2023
text2text-generation

Introduction of FRED-T5-1.7B

Model Details of FRED-T5-1.7B

FRED-T5 1.7B (Full-scale Russian Enhanced Denoisers T5)

The model architecture design, pretraining, and evaluation are documented in our preprint: A Family of Pretrained Transformer Language Models for Russian .

The model was trained by SberDevices .

Architecture based on T5.

It has 24 layers and 1536 hidden size. More details in config.json.

The model trained on a mixture of 7 denoisers like UL2 with several differences ( https://arxiv.org/abs/2205.05131 ).

It was trained on Russian language corpus (300GB). The dataset is the same as for ruT5 models.

Bbpe tokenizer. 50257 + special tokens 107. Prefix tokens: '<LM>', '<SC1>',.. '<SC6>'

First half of the time model trained on the small part of all dataset (1%,3GB) and without prefixes in each task.

For RSG, we trained as described in the T5 paper. First, we trained multitask for all tasks. Then we took the best checkpoint for the task and trained it further. RSG submit here https://russiansuperglue.com/login/submit_info/1936

Total training time was around 45 days on 112 A100 GPUs.

Usage (HuggingFace Models Repository)
import torch
from transformers import GPT2Tokenizer, T5ForConditionalGeneration 
tokenizer = GPT2Tokenizer.from_pretrained('ai-forever/FRED-T5-1.7B',eos_token='</s>')
model = T5ForConditionalGeneration.from_pretrained('ai-forever/FRED-T5-1.7B')
device='cuda'
model.to(device)

#Prefix <LM>
lm_text='<LM>Принялся Кутузов рассказывать свою историю как он сюда попал. Началось'
input_ids=torch.tensor([tokenizer.encode(lm_text)]).to(device)
outputs=model.generate(input_ids,eos_token_id=tokenizer.eos_token_id,early_stopping=True)
print(tokenizer.decode(outputs[0][1:]))

# print result: с того, что он был в армии, служил в артиллерии</s>.

#Prefix <SC1>
lm_text='<SC1>Принялся Кутузов рассказывать свою историю <extra_id_0>. Началось с того, что он был в армии, служил в артиллерии.'
input_ids=torch.tensor([tokenizer.encode(lm_text)]).to(device)
outputs=model.generate(input_ids,eos_token_id=tokenizer.eos_token_id,early_stopping=True)
print(tokenizer.decode(outputs[0][1:]))

#print result: '<extra_id_0>, как он воевал</s>'

# Prefix <SC5> 
lm_text='<SC5>Принялся Кутузов рассказывать свою историю <extra_id_0>. Началось с того, что он был в армии, служил в артиллерии.'
input_ids=torch.tensor([tokenizer.encode(lm_text)]).to(device)
outputs=model.generate(input_ids,eos_token_id=tokenizer.eos_token_id,early_stopping=True)
tokenizer.decode(outputs[0][1:])

#print result: '<extra_id_0>, как он стал генералом</s>'

Authors

  • NLP core team RnD Telegram channel :
    • Dmitry Zmitrovich
    • Andrei Kalmykov
    • Vitaly Kadulin
    • Mikhail Novikov
    • Alexey Khoroshilov

Salute AI Community .

Cite us

@misc{zmitrovich2023family,
      title={A Family of Pretrained Transformer Language Models for Russian}, 
      author={Dmitry Zmitrovich and Alexander Abramov and Andrey Kalmykov and Maria Tikhonova and Ekaterina Taktasheva and Danil Astafurov and Mark Baushenko and Artem Snegirev and Tatiana Shavrina and Sergey Markov and Vladislav Mikhailov and Alena Fenogenova},
      year={2023},
      eprint={2309.10931},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Runs of ai-forever FRED-T5-1.7B on huggingface.co

2.2K
Total runs
-10
24-hour runs
-24
3-day runs
351
7-day runs
716
30-day runs

More Information About FRED-T5-1.7B huggingface.co Model

More FRED-T5-1.7B license Visit here:

https://choosealicense.com/licenses/apache-2.0

FRED-T5-1.7B huggingface.co

FRED-T5-1.7B huggingface.co is an AI model on huggingface.co that provides FRED-T5-1.7B's model effect (), which can be used instantly with this ai-forever FRED-T5-1.7B model. huggingface.co supports a free trial of the FRED-T5-1.7B model, and also provides paid use of the FRED-T5-1.7B. Support call FRED-T5-1.7B model through api, including Node.js, Python, http.

ai-forever FRED-T5-1.7B online free

FRED-T5-1.7B huggingface.co is an online trial and call api platform, which integrates FRED-T5-1.7B's modeling effects, including api services, and provides a free online trial of FRED-T5-1.7B, you can try FRED-T5-1.7B online for free by clicking the link below.

ai-forever FRED-T5-1.7B online free url in huggingface.co:

https://huggingface.co/ai-forever/FRED-T5-1.7B

FRED-T5-1.7B install

FRED-T5-1.7B is an open source model from GitHub that offers a free installation service, and any user can find FRED-T5-1.7B on GitHub to install. At the same time, huggingface.co provides the effect of FRED-T5-1.7B install, users can directly use FRED-T5-1.7B installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

FRED-T5-1.7B install url in huggingface.co:

https://huggingface.co/ai-forever/FRED-T5-1.7B

Url of FRED-T5-1.7B

FRED-T5-1.7B huggingface.co Url

Provider of FRED-T5-1.7B huggingface.co

ai-forever
ORGANIZATIONS

Other API from ai-forever

huggingface.co

Total runs: 525.5K
Run Growth: 507.4K
Growth Rate: 96.56%
Updated: 11月 03 2023
huggingface.co

Total runs: 10.6K
Run Growth: 1.5K
Growth Rate: 13.75%
Updated: 12月 05 2023
huggingface.co

Total runs: 8.2K
Run Growth: 5.1K
Growth Rate: 59.45%
Updated: 12月 29 2024
huggingface.co

Total runs: 5.9K
Run Growth: 3.5K
Growth Rate: 60.26%
Updated: 12月 11 2023
huggingface.co

Total runs: 2.3K
Run Growth: -408
Growth Rate: -17.86%
Updated: 12月 05 2023
huggingface.co

Total runs: 315
Run Growth: 158
Growth Rate: 50.16%
Updated: 1月 26 2023
huggingface.co

Total runs: 0
Run Growth: 0
Growth Rate: 0.00%
Updated: 12月 24 2021
huggingface.co

Total runs: 0
Run Growth: 0
Growth Rate: 0.00%
Updated: 6月 08 2023
huggingface.co

Total runs: 0
Run Growth: 0
Growth Rate: 0.00%
Updated: 9月 21 2021