Total runs:
5.6M
Run Growth:
2.7M
Growth Rate:
47.46%
allenai / OLMoE-1B-7B-0924-SFT
Model's Last Updated: Septiembre 04 2024
Introduction of OLMoE-1B-7B-0924-SFT
Model Details of OLMoE-1B-7B-0924-SFT
Model Summary
This model is an intermediate training checkpoint during post-training, after the Supervised Fine-Tuning (SFT) step. For best performance, we recommend you use the OLMoE-Instruct version.
- Paper : https://arxiv.org/abs/2409.02060
- Pretraining Checkpoints , Code , Data and Logs .
- SFT (Supervised Fine-Tuning) Checkpoints , Code , Data and Logs .
- DPO/KTO (Direct Preference Optimization/Kahneman-Tversky Optimization) , Checkpoints , Preference Data , DPO code , KTO code and Logs .
Branches:
-
main: Instruction tuned / supervised finetuned (SFT) model of https://hf.co/allenai/OLMoE-1B-7B-0924 (mainbranch) -
load-balancing: Ablation with load balancing loss during SFT -
non-annealed: Ablation starting from the checkpoint prior to annealing (branchstep1200000-tokens5033Bof https://hf.co/allenai/OLMoE-1B-7B-0924 ) rather than the annealed checkpoint (branchmainof https://hf.co/allenai/OLMoE-1B-7B-0924 )
Citation
@misc{muennighoff2024olmoeopenmixtureofexpertslanguage,
title={OLMoE: Open Mixture-of-Experts Language Models},
author={Niklas Muennighoff and Luca Soldaini and Dirk Groeneveld and Kyle Lo and Jacob Morrison and Sewon Min and Weijia Shi and Pete Walsh and Oyvind Tafjord and Nathan Lambert and Yuling Gu and Shane Arora and Akshita Bhagia and Dustin Schwenk and David Wadden and Alexander Wettig and Binyuan Hui and Tim Dettmers and Douwe Kiela and Ali Farhadi and Noah A. Smith and Pang Wei Koh and Amanpreet Singh and Hannaneh Hajishirzi},
year={2024},
eprint={2409.02060},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2409.02060},
}
Runs of allenai OLMoE-1B-7B-0924-SFT on huggingface.co
103
Total runs
0
24-hour runs
-118
3-day runs
-146
7-day runs
-188
30-day runs
More Information About OLMoE-1B-7B-0924-SFT huggingface.co Model
Url of OLMoE-1B-7B-0924-SFT
OLMoE-1B-7B-0924-SFT huggingface.co Url
Provider of OLMoE-1B-7B-0924-SFT huggingface.co
Other API from allenai
Updated: Abril 05 2023
Total runs:
803.5K
Run Growth:
118.1K
Growth Rate:
14.70%
Total runs:
173.9K
Run Growth:
-113.8K
Growth Rate:
-65.44%
Total runs:
134.2K
Run Growth:
110.4K
Growth Rate:
82.06%
Total runs:
94.4K
Run Growth:
-2.1K
Growth Rate:
-2.24%
Total runs:
91.3K
Run Growth:
63.3K
Growth Rate:
69.57%
Total runs:
77.3K
Run Growth:
-517.1K
Growth Rate:
-669.13%
Total runs:
63.3K
Run Growth:
51.7K
Growth Rate:
81.63%
Total runs:
61.6K
Run Growth:
-50.5K
Growth Rate:
-81.96%
Total runs:
49.7K
Run Growth:
-640
Growth Rate:
-1.29%
Total runs:
41.2K
Run Growth:
41.2K
Growth Rate:
99.97%
Total runs:
35.6K
Run Growth:
26.6K
Growth Rate:
98.51%
Total runs:
31.2K
Run Growth:
-19.4K
Growth Rate:
-62.31%
Total runs:
27.7K
Run Growth:
24.5K
Growth Rate:
88.59%
Total runs:
23.4K
Run Growth:
1.4K
Growth Rate:
5.61%
Total runs:
22.6K
Run Growth:
-54
Growth Rate:
-0.24%
Total runs:
21.2K
Run Growth:
-18.4K
Growth Rate:
-87.09%
Total runs:
17.9K
Run Growth:
-22.8K
Growth Rate:
-127.30%
Total runs:
16.3K
Run Growth:
5.4K
Growth Rate:
27.42%
Total runs:
12.4K
Run Growth:
8.0K
Growth Rate:
69.22%
Total runs:
11.5K
Run Growth:
4.9K
Growth Rate:
42.53%
Total runs:
11.3K
Run Growth:
977
Growth Rate:
8.66%
Total runs:
10.5K
Run Growth:
2.7K
Growth Rate:
25.33%
Total runs:
10.5K
Run Growth:
43
Growth Rate:
0.41%
Total runs:
7.6K
Run Growth:
7.9K
Growth Rate:
65.67%
Total runs:
7.1K
Run Growth:
-19.8K
Growth Rate:
-277.09%
Total runs:
6.9K
Run Growth:
880
Growth Rate:
12.71%
Total runs:
6.5K
Run Growth:
-2.3K
Growth Rate:
-34.31%
Total runs:
6.4K
Run Growth:
3.3K
Growth Rate:
51.01%
Total runs:
4.9K
Run Growth:
2.1K
Growth Rate:
45.30%
Total runs:
4.7K
Run Growth:
2.7K
Growth Rate:
56.97%
Total runs:
4.5K
Run Growth:
4.4K
Growth Rate:
97.96%
Total runs:
4.3K
Run Growth:
3.2K
Growth Rate:
74.66%
Total runs:
4.1K
Run Growth:
-888
Growth Rate:
-21.90%
Total runs:
3.3K
Run Growth:
3.1K
Growth Rate:
93.97%
Total runs:
2.4K
Run Growth:
-2.0K
Growth Rate:
-77.45%
Total runs:
2.3K
Run Growth:
1.6K
Growth Rate:
69.43%
Total runs:
2.3K
Run Growth:
2.2K
Growth Rate:
98.27%
Total runs:
2.1K
Run Growth:
2.2K
Growth Rate:
33.30%
Total runs:
2.1K
Run Growth:
578
Growth Rate:
34.24%
Total runs:
2.0K
Run Growth:
-2.5K
Growth Rate:
-123.46%
Total runs:
1.9K
Run Growth:
854
Growth Rate:
44.90%
Total runs:
1.9K
Run Growth:
1.9K
Growth Rate:
99.26%
Total runs:
1.9K
Run Growth:
862
Growth Rate:
45.37%
Total runs:
1.9K
Run Growth:
1.5K
Growth Rate:
82.01%
Total runs:
1.9K
Run Growth:
1.8K
Growth Rate:
97.27%
Total runs:
1.8K
Run Growth:
162
Growth Rate:
9.02%
Total runs:
1.7K
Run Growth:
-160
Growth Rate:
-9.16%
Total runs:
1.7K
Run Growth:
943
Growth Rate:
72.21%
Total runs:
1.6K
Run Growth:
885
Growth Rate:
47.40%
Total runs:
1.5K
Run Growth:
824
Growth Rate:
53.93%
Total runs:
1.4K
Run Growth:
841
Growth Rate:
57.17%
Total runs:
1.0K
Run Growth:
-34
Growth Rate:
-3.30%
Total runs:
946
Run Growth:
-3.2K
Growth Rate:
-239.15%
Total runs:
936
Run Growth:
513
Growth Rate:
54.81%
Total runs:
920
Run Growth:
854
Growth Rate:
92.83%
Total runs:
873
Run Growth:
-148
Growth Rate:
-16.95%
Total runs:
842
Run Growth:
839
Growth Rate:
99.64%
Total runs:
669
Run Growth:
392
Growth Rate:
58.59%
Total runs:
667
Run Growth:
237
Growth Rate:
35.53%
Total runs:
655
Run Growth:
323
Growth Rate:
43.01%
Total runs:
542
Run Growth:
-250
Growth Rate:
-46.13%
Total runs:
522
Run Growth:
-340
Growth Rate:
-65.13%
Total runs:
469
Run Growth:
134
Growth Rate:
29.00%
Total runs:
406
Run Growth:
383
Growth Rate:
94.33%
Total runs:
382
Run Growth:
333
Growth Rate:
87.17%
Total runs:
360
Run Growth:
13
Growth Rate:
4.30%
Total runs:
329
Run Growth:
242
Growth Rate:
73.11%
Total runs:
326
Run Growth:
-125
Growth Rate:
-38.34%
Total runs:
317
Run Growth:
-5.8K
Growth Rate:
-1837.22%
Total runs:
299
Run Growth:
36
Growth Rate:
12.08%
Total runs:
284
Run Growth:
190
Growth Rate:
64.41%
Total runs:
254
Run Growth:
176
Growth Rate:
69.29%
Total runs:
218
Run Growth:
204
Growth Rate:
93.58%
Total runs:
194
Run Growth:
22
Growth Rate:
11.46%
Total runs:
189
Run Growth:
-216
Growth Rate:
-108.00%
Total runs:
179
Run Growth:
-41
Growth Rate:
-22.91%
Total runs:
177
Run Growth:
162
Growth Rate:
91.53%