The model was pretrained with sequence length 1024 using transformers by the
SberDevices
team on 80B tokens around 3 epochs. After that, the model was finetuned with the context size of 2048.
Total training time took around one week on 32 GPUs.
@misc{zmitrovich2023family,
title={A Family of Pretrained Transformer Language Models for Russian},
author={Dmitry Zmitrovich and Alexander Abramov and Andrey Kalmykov and Maria Tikhonova and Ekaterina Taktasheva and Danil Astafurov and Mark Baushenko and Artem Snegirev and Tatiana Shavrina and Sergey Markov and Vladislav Mikhailov and Alena Fenogenova},
year={2023},
eprint={2309.10931},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
Runs of ai-forever rugpt3small_based_on_gpt2 on huggingface.co
24.7K
Total runs
0
24-hour runs
-87
3-day runs
1.2K
7-day runs
6.1K
30-day runs
More Information About rugpt3small_based_on_gpt2 huggingface.co Model
rugpt3small_based_on_gpt2 huggingface.co
rugpt3small_based_on_gpt2 huggingface.co is an AI model on huggingface.co that provides rugpt3small_based_on_gpt2's model effect (), which can be used instantly with this ai-forever rugpt3small_based_on_gpt2 model. huggingface.co supports a free trial of the rugpt3small_based_on_gpt2 model, and also provides paid use of the rugpt3small_based_on_gpt2. Support call rugpt3small_based_on_gpt2 model through api, including Node.js, Python, http.
rugpt3small_based_on_gpt2 huggingface.co is an online trial and call api platform, which integrates rugpt3small_based_on_gpt2's modeling effects, including api services, and provides a free online trial of rugpt3small_based_on_gpt2, you can try rugpt3small_based_on_gpt2 online for free by clicking the link below.
ai-forever rugpt3small_based_on_gpt2 online free url in huggingface.co:
rugpt3small_based_on_gpt2 is an open source model from GitHub that offers a free installation service, and any user can find rugpt3small_based_on_gpt2 on GitHub to install. At the same time, huggingface.co provides the effect of rugpt3small_based_on_gpt2 install, users can directly use rugpt3small_based_on_gpt2 installed effect in huggingface.co for debugging and trial. It also supports api for free installation.
rugpt3small_based_on_gpt2 install url in huggingface.co: