ai-forever / mGPT

huggingface.co
Total runs: 10.6K
24-hour runs: -61
7-day runs: -114
30-day runs: 1.5K
Model's Last Updated: December 05 2023
text-generation

Introduction of mGPT

Model Details of mGPT

Multilingual GPT model

We introduce a family of autoregressive GPT-like models with 1.3 billion parameters trained on 61 languages from 25 language families using Wikipedia and Colossal Clean Crawled Corpus.

We reproduce the GPT-3 architecture using GPT-2 sources and the sparse attention mechanism, Deepspeed and Megatron frameworks allows us to effectively parallelize the training and inference steps. The resulting models show performance on par with the recently released XGLM models at the same time covering more languages and enhancing NLP possibilities for low resource languages.

Code

The source code for the mGPT XL model is available on Github

Paper

mGPT: Few-Shot Learners Go Multilingual

Abstract PDF

@misc{https://doi.org/10.48550/arxiv.2204.07580,
 doi = {10.48550/ARXIV.2204.07580},
 
 url = {https://arxiv.org/abs/2204.07580},
 
 author = {Shliazhko, Oleh and Fenogenova, Alena and Tikhonova, Maria and Mikhailov, Vladislav and Kozlova, Anastasia and Shavrina, Tatiana},
 
 keywords = {Computation and Language (cs.CL), Artificial Intelligence (cs.AI), FOS: Computer and information sciences, FOS: Computer and information sciences, I.2; I.2.7, 68-06, 68-04, 68T50, 68T01},
 
 title = {mGPT: Few-Shot Learners Go Multilingual},
 
 publisher = {arXiv},
 
 year = {2022},
 
 copyright = {Creative Commons Attribution 4.0 International}
}
Languages

Model supports 61 languages:

ISO codes: ar he vi id jv ms tl lv lt eu ml ta te hy bn mr hi ur af da en de sv fr it pt ro es el os tg fa ja ka ko th bxr xal mn sw yo be bg ru uk pl my uz ba kk ky tt az cv tr tk tyv sax et fi hu

Languages:

Arabic, Hebrew, Vietnamese, Indonesian, Javanese, Malay, Tagalog, Latvian, Lithuanian, Basque, Malayalam, Tamil, Telugu, Armenian, Bengali, Marathi, Hindi, Urdu, Afrikaans, Danish, English, German, Swedish, French, Italian, Portuguese, Romanian, Spanish, Greek, Ossetian, Tajik, Persian, Japanese, Georgian, Korean, Thai, Buryat, Kalmyk, Mongolian, Swahili, Yoruba, Belarusian, Bulgarian, Russian, Ukrainian, Polish, Burmese, Uzbek, Bashkir, Kazakh, Kyrgyz, Tatar, Azerbaijani, Chuvash, Turkish, Turkmen, Tuvan, Yakut, Estonian, Finnish, Hungarian

Training Data Statistics
  • Size: 488 Billion UTF characters
"General training corpus statistics"
Details

The model was trained with sequence length 512 using Megatron and Deepspeed libs by SberDevices team on a dataset of 600 GB of texts in 61 languages. The model has seen 440 billion BPE tokens in total.

Total training time was around 14 days on 256 Nvidia V100 GPUs.

Runs of ai-forever mGPT on huggingface.co

10.6K
Total runs
-61
24-hour runs
56
3-day runs
-114
7-day runs
1.5K
30-day runs

More Information About mGPT huggingface.co Model

mGPT huggingface.co

mGPT huggingface.co is an AI model on huggingface.co that provides mGPT's model effect (), which can be used instantly with this ai-forever mGPT model. huggingface.co supports a free trial of the mGPT model, and also provides paid use of the mGPT. Support call mGPT model through api, including Node.js, Python, http.

ai-forever mGPT online free

mGPT huggingface.co is an online trial and call api platform, which integrates mGPT's modeling effects, including api services, and provides a free online trial of mGPT, you can try mGPT online for free by clicking the link below.

ai-forever mGPT online free url in huggingface.co:

https://huggingface.co/ai-forever/mGPT

mGPT install

mGPT is an open source model from GitHub that offers a free installation service, and any user can find mGPT on GitHub to install. At the same time, huggingface.co provides the effect of mGPT install, users can directly use mGPT installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

mGPT install url in huggingface.co:

https://huggingface.co/ai-forever/mGPT

Url of mGPT

Provider of mGPT huggingface.co

ai-forever
ORGANIZATIONS

Other API from ai-forever

huggingface.co

Total runs: 525.5K
Run Growth: 507.4K
Growth Rate: 96.56%
Updated: November 03 2023
huggingface.co

Total runs: 8.2K
Run Growth: 5.1K
Growth Rate: 59.45%
Updated: December 29 2024
huggingface.co

Total runs: 5.9K
Run Growth: 3.5K
Growth Rate: 60.26%
Updated: December 11 2023
huggingface.co

Total runs: 2.3K
Run Growth: -408
Growth Rate: -17.86%
Updated: December 05 2023
huggingface.co

Total runs: 1.3K
Run Growth: -6.3K
Growth Rate: -493.39%
Updated: December 05 2023
huggingface.co

Total runs: 315
Run Growth: 158
Growth Rate: 50.16%
Updated: January 26 2023
huggingface.co

Total runs: 0
Run Growth: 0
Growth Rate: 0.00%
Updated: December 24 2021
huggingface.co

Total runs: 0
Run Growth: 0
Growth Rate: 0.00%
Updated: June 08 2023
huggingface.co

Total runs: 0
Run Growth: 0
Growth Rate: 0.00%
Updated: September 21 2021