ai-forever / mGPT-armenian

huggingface.co
Total runs: 188
24-hour runs: 4
7-day runs: 10
30-day runs: 132
Model's Last Updated: Août 31 2022
text-generation

Introduction of mGPT-armenian

Model Details of mGPT-armenian

Multilingual GPT model, Armenian language finetune

We introduce a monolingual GPT-3-based model for Armenian language

The model is based on mGPT , a family of autoregressive GPT-like models with 1.3 billion parameters trained on 60 languages from 25 language families using Wikipedia and Colossal Clean Crawled Corpus.

We reproduce the GPT-3 architecture using GPT-2 sources and the sparse attention mechanism, Deepspeed and Megatron frameworks allows us to effectively parallelize the training and inference steps. The resulting models show performance on par with the recently released XGLM models at the same time covering more languages and enhancing NLP possibilities for low resource languages.

Code

The source code for the mGPT XL model is available on Github

Paper

mGPT: Few-Shot Learners Go Multilingual

Abstract PDF

@misc{https://doi.org/10.48550/arxiv.2204.07580,
 doi = {10.48550/ARXIV.2204.07580},
 url = {https://arxiv.org/abs/2204.07580},
 author = {Shliazhko, Oleh and Fenogenova, Alena and Tikhonova, Maria and Mikhailov, Vladislav and Kozlova, Anastasia and Shavrina, Tatiana},
 keywords = {Computation and Language (cs.CL), Artificial Intelligence (cs.AI), FOS: Computer and information sciences, FOS: Computer and information sciences, I.2; I.2.7, 68-06, 68-04, 68T50, 68T01},
 title = {mGPT: Few-Shot Learners Go Multilingual},
 publisher = {arXiv},
 year = {2022},
 copyright = {Creative Commons Attribution 4.0 International}
}
Training

The model was fine-tuned on 170GB of Armenian texts, including MC4, Archive.org fiction, EANC public data, OpenSubtitles, OSCAR corpus and blog texts.

Val perplexity is 2.046.

The mGPT model was pre-trained for 12 days x 256 GPU (Tesla NVidia V100), 4 epochs, then 9 days x 64 GPU, 1 epoch

The Armenian finetune was around 7 days with 4 Tesla NVidia V100 and has made 160k steps.

What happens on this image? The model is originally trained with sparse attention masks, then fine-tuned with no sparsity on the last steps (perplexity and loss peak). Getting rid of the sparsity in the end of the training helps to integrate the model into the GPT2 HF class.

Runs of ai-forever mGPT-armenian on huggingface.co

188
Total runs
4
24-hour runs
9
3-day runs
10
7-day runs
132
30-day runs

More Information About mGPT-armenian huggingface.co Model

More mGPT-armenian license Visit here:

https://choosealicense.com/licenses/apache-2.0

mGPT-armenian huggingface.co

mGPT-armenian huggingface.co is an AI model on huggingface.co that provides mGPT-armenian's model effect (), which can be used instantly with this ai-forever mGPT-armenian model. huggingface.co supports a free trial of the mGPT-armenian model, and also provides paid use of the mGPT-armenian. Support call mGPT-armenian model through api, including Node.js, Python, http.

ai-forever mGPT-armenian online free

mGPT-armenian huggingface.co is an online trial and call api platform, which integrates mGPT-armenian's modeling effects, including api services, and provides a free online trial of mGPT-armenian, you can try mGPT-armenian online for free by clicking the link below.

ai-forever mGPT-armenian online free url in huggingface.co:

https://huggingface.co/ai-forever/mGPT-armenian

mGPT-armenian install

mGPT-armenian is an open source model from GitHub that offers a free installation service, and any user can find mGPT-armenian on GitHub to install. At the same time, huggingface.co provides the effect of mGPT-armenian install, users can directly use mGPT-armenian installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

mGPT-armenian install url in huggingface.co:

https://huggingface.co/ai-forever/mGPT-armenian

Url of mGPT-armenian

mGPT-armenian huggingface.co Url

Provider of mGPT-armenian huggingface.co

ai-forever
ORGANIZATIONS

Other API from ai-forever

huggingface.co

Total runs: 525.5K
Run Growth: 507.4K
Growth Rate: 96.56%
Updated: Novembre 03 2023
huggingface.co

Total runs: 10.6K
Run Growth: 1.5K
Growth Rate: 13.75%
Updated: Décembre 05 2023
huggingface.co

Total runs: 8.2K
Run Growth: 5.1K
Growth Rate: 59.45%
Updated: Décembre 29 2024
huggingface.co

Total runs: 5.9K
Run Growth: 3.5K
Growth Rate: 60.26%
Updated: Décembre 11 2023
huggingface.co

Total runs: 2.3K
Run Growth: -408
Growth Rate: -17.86%
Updated: Décembre 05 2023
huggingface.co

Total runs: 1.8K
Run Growth: 5
Growth Rate: 0.28%
Updated: Décembre 28 2023
huggingface.co

Total runs: 1.3K
Run Growth: -6.3K
Growth Rate: -493.39%
Updated: Décembre 05 2023
huggingface.co

Total runs: 315
Run Growth: 158
Growth Rate: 50.16%
Updated: Janvier 26 2023
huggingface.co

Total runs: 0
Run Growth: 0
Growth Rate: 0.00%
Updated: Décembre 24 2021
huggingface.co

Total runs: 0
Run Growth: 0
Growth Rate: 0.00%
Updated: Juin 08 2023
huggingface.co

Total runs: 0
Run Growth: 0
Growth Rate: 0.00%
Updated: Septembre 21 2021