ai-forever / mGPT-13B

huggingface.co
Total runs: 2.3K
24-hour runs: 24
7-day runs: -97
30-day runs: -360
Model's Last Updated: December 05 2023
text-generation

Introduction of mGPT-13B

Model Details of mGPT-13B

🌻 mGPT 13B

Multilingual language model. This model was trained on the 61 languages from 25 language families (see the list below).

Dataset

Model was pretrained on a 600Gb of texts, mostly from MC4 and Wikipedia. Training data was deduplicated, the text deduplication includes 64-bit hashing of each text in the corpus for keeping texts with a unique hash. We also filter the documents based on their text compression rate using zlib4. The most strongly and weakly compressing deduplicated texts are discarded.

Here is the table with number of tokens for each language in the pretraining corpus on a logarithmic scale:

Languages

Afrikaans (af), Arabic (ar), Armenian (hy), Azerbaijani (az), Basque (eu), Bashkir (ba), Belarusian (be), Bengali (bn), Bulgarian (bg), Burmese (my), Buryat (bxr), Chuvash (cv), Danish (da), English (en), Estonian (et), Finnish (fi), French (fr), Georgian (ka), German (de), Greek (el), Hebrew (he), Hindi (hi), Hungarian (hu), Indonesian (id), Italian (it), Japanese (ja), Javanese (jv), Kalmyk (xal), Kazakh (kk), Korean (ko), Kyrgyz (ky), Latvian (lv), Lithuanian (lt), Malay (ms), Malayalam (ml), Marathi (mr), Mongolian (mn), Ossetian (os), Persian (fa), Polish (pl), Portuguese (pt), Romanian (ro), Russian (ru), Spanish (es), Swedish (sv), Swahili (sw), Tatar (tt), Telugu (te), Thai (th), Turkish (tr), Turkmen (tk), Tuvan (tyv), Ukrainian (uk), Uzbek (uz), Vietnamese (vi), Yakut (sax), Yoruba (yo)

By language family
Language Family Languages
Afro-Asiatic Arabic (ar), Hebrew (he)
Austro-Asiatic Vietnamese (vi)
Austronesian Indonesian (id), Javanese (jv), Malay (ms), Tagalog (tl)
Baltic Latvian (lv), Lithuanian (lt)
Basque Basque (eu)
Dravidian Malayalam (ml), Tamil (ta), Telugu (te)
Indo-European (Armenian) Armenian (hy)
Indo-European (Indo-Aryan) Bengali (bn), Marathi (mr), Hindi (hi), Urdu (ur)
Indo-European (Germanic) Afrikaans (af), Danish (da), English (en), German (de), Swedish (sv)
Indo-European (Romance) French (fr), Italian (it), Portuguese (pt), Romanian (ro), Spanish (es)
Indo-European (Greek) Greek (el)
Indo-European (Iranian) Ossetian (os), Tajik (tg), Persian (fa)
Japonic Japanese (ja)
Kartvelian Georgian (ka)
Koreanic Korean (ko)
Kra-Dai Thai (th)
Mongolic Buryat (bxr), Kalmyk (xal), Mongolian (mn)
Niger-Congo Swahili (sw), Yoruba (yo)
Slavic Belarusian (be), Bulgarian (bg), Russian (ru), Ukrainian (uk), Polish (pl)
Sino-Tibetan Burmese (my)
Turkic (Karluk) Uzbek (uz)
Turkic (Kipchak) Bashkir (ba), Kazakh (kk), Kyrgyz (ky), Tatar (tt)
Turkic (Oghuz) Azerbaijani (az), Chuvash (cv), Turkish (tr), Turkmen (tk)
Turkic (Siberian) Tuvan (tyv), Yakut (sax)
Uralic Estonian (et), Finnish (fi), Hungarian (hu)
Technical details

The models are pretrained on 16 V100 GPUs for 600k training steps with a set of fixed hyperparameters: vocabulary size of 100k, context window of 2048, learning rate of 2e−4, and batch size of 4.

The mGPT architecture is based on GPT-3. We use the architecture description by Brown et al., the code base on GPT-2 (Radford et al., 2019) in the HuggingFace library (Wolf et al., 2020) and Megatron-LM (Shoeybi et al., 2019).

Perplexity

The mGPT13B model achieves the best perplexities within the 2-to-10 score range for the majority of languages, including Dravidian (Malayalam, Tamil, Telugu), Indo-Aryan (Bengali, Hindi, Marathi), Slavic (Belarusian, Ukrainian, Russian, Bulgarian), Sino-Tibetan (Burmese), Kipchak (Bashkir, Kazakh) and others. Higher perplexities up to 20 are for only seven languages from different families.

Language-wise perplexity results

Family-wise perplexity results

The scores are averaged over the number of languages within each family.

Runs of ai-forever mGPT-13B on huggingface.co

2.3K
Total runs
24
24-hour runs
-7
3-day runs
-97
7-day runs
-360
30-day runs

More Information About mGPT-13B huggingface.co Model

More mGPT-13B license Visit here:

https://choosealicense.com/licenses/mit

mGPT-13B huggingface.co

mGPT-13B huggingface.co is an AI model on huggingface.co that provides mGPT-13B's model effect (), which can be used instantly with this ai-forever mGPT-13B model. huggingface.co supports a free trial of the mGPT-13B model, and also provides paid use of the mGPT-13B. Support call mGPT-13B model through api, including Node.js, Python, http.

ai-forever mGPT-13B online free

mGPT-13B huggingface.co is an online trial and call api platform, which integrates mGPT-13B's modeling effects, including api services, and provides a free online trial of mGPT-13B, you can try mGPT-13B online for free by clicking the link below.

ai-forever mGPT-13B online free url in huggingface.co:

https://huggingface.co/ai-forever/mGPT-13B

mGPT-13B install

mGPT-13B is an open source model from GitHub that offers a free installation service, and any user can find mGPT-13B on GitHub to install. At the same time, huggingface.co provides the effect of mGPT-13B install, users can directly use mGPT-13B installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

mGPT-13B install url in huggingface.co:

https://huggingface.co/ai-forever/mGPT-13B

Url of mGPT-13B

Provider of mGPT-13B huggingface.co

ai-forever
ORGANIZATIONS

Other API from ai-forever

huggingface.co

Total runs: 525.5K
Run Growth: 485.9K
Growth Rate: 96.59%
Updated: November 03 2023
huggingface.co

Total runs: 10.6K
Run Growth: 1.6K
Growth Rate: 14.99%
Updated: December 05 2023
huggingface.co

Total runs: 8.2K
Run Growth: 5.1K
Growth Rate: 59.45%
Updated: December 29 2024
huggingface.co

Total runs: 5.9K
Run Growth: 3.6K
Growth Rate: 61.73%
Updated: December 11 2023
huggingface.co

Total runs: 1.8K
Run Growth: 78
Growth Rate: 4.35%
Updated: December 28 2023
huggingface.co

Total runs: 315
Run Growth: 165
Growth Rate: 52.05%
Updated: January 26 2023
huggingface.co

Total runs: 0
Run Growth: 0
Growth Rate: 0.00%
Updated: December 24 2021
huggingface.co

Total runs: 0
Run Growth: 0
Growth Rate: 0.00%
Updated: June 08 2023
huggingface.co

Total runs: 0
Run Growth: 0
Growth Rate: 0.00%
Updated: September 21 2021