Total runs:
5.6M
Run Growth:
2.7M
Growth Rate:
47.46%
allenai / llama-3.1-tulu-2-8b-uf-mean-rm
Model's Last Updated: Août 15 2024
Introduction of llama-3.1-tulu-2-8b-uf-mean-rm
Model Details of llama-3.1-tulu-2-8b-uf-mean-rm
Model Card for Llama 3.1 Tulu V2 8B RM - UltraFeedback
Tulu is a series of language models that are trained to act as helpful assistants. This is a 8B reward model used for PPO training trained on the UltraFeedback dataset.
For more details, read the paper: Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback .
Note this model is finetuned from Llama 3.1, released under the Meta Llama 3.1 community license, included here under
llama_3_license.txt
.
Performance
We evaluate the model on RewardBench :
| Model | Score | Chat | Chat Hard | Safety | Reasoning |
|---|---|---|---|---|---|
| Llama 3.1 Tulu 2 8b UF RM (this model) | 73.3 | 98.0 | 59.6 | 60.6 | 74.7 |
| Llama 3.1 Tulu 2 70b UF RM |
Model description
- Model type: A reward model trained on UltraFeedback, designed to be used in RLHF training.
- Language(s) (NLP): English
- License: Apache 2.0.
- Finetuned from model: allenai/llama-3.1-tulu-2-8b
Model Sources
- Repository: https://github.com/allenai/open-instruct
-
Dataset:
Data used to train this model can be found
here
- specifically the
ultrafeedback_mean_aspectssplit.
Input Format
The model is trained to use the following format (note the newlines):
<|user|>
Your message here!
<|assistant|>
For best results, format all inputs in this manner.
Make sure to include a newline after
<|assistant|>
, this can affect generation quality quite a bit.
We have included a
chat template
in the tokenizer implementing this template.
Intended uses & limitations
The model was initially fine-tuned on a filtered and preprocessed of the Tulu V2 mix dataset , which contains a diverse range of human created instructions and synthetic dialogues generated primarily by other LLMs. We then further trained the model with a Jax RM trainer built on EasyLM on the dataset mentioned above. This model is meant as a research artefact.
Training hyperparameters
The following hyperparameters were used during RM training:
- learning_rate: 5e-06
- total_train_batch_size: 64
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear cooldown to 0.
- lr_scheduler_warmup_ratio: 0.03
- num_epochs: 1.0
Citation
If you find Tulu 2.5 is useful in your work, please cite it with:
@misc{ivison2024unpacking,
title={{Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback}},
author={{Hamish Ivison and Yizhong Wang and Jiacheng Liu and Ellen Wu and Valentina Pyatkin and Nathan Lambert and Yejin Choi and Noah A. Smith and Hannaneh Hajishirzi}}
year={2024},
eprint={2406.09279},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
Runs of allenai llama-3.1-tulu-2-8b-uf-mean-rm on huggingface.co
179
Total runs
0
24-hour runs
-93
3-day runs
-104
7-day runs
-41
30-day runs
More Information About llama-3.1-tulu-2-8b-uf-mean-rm huggingface.co Model
Url of llama-3.1-tulu-2-8b-uf-mean-rm
llama-3.1-tulu-2-8b-uf-mean-rm huggingface.co Url
Provider of llama-3.1-tulu-2-8b-uf-mean-rm huggingface.co
Other API from allenai
Updated: Avril 05 2023
Total runs:
803.5K
Run Growth:
118.1K
Growth Rate:
14.70%
Total runs:
173.9K
Run Growth:
-113.8K
Growth Rate:
-65.44%
Total runs:
134.2K
Run Growth:
110.4K
Growth Rate:
82.06%
Total runs:
94.4K
Run Growth:
-2.1K
Growth Rate:
-2.24%
Total runs:
91.3K
Run Growth:
63.3K
Growth Rate:
69.57%
Total runs:
77.3K
Run Growth:
-517.1K
Growth Rate:
-669.13%
Total runs:
75.6K
Run Growth:
4.4K
Growth Rate:
5.77%
Total runs:
63.3K
Run Growth:
51.7K
Growth Rate:
81.63%
Total runs:
61.6K
Run Growth:
-50.5K
Growth Rate:
-81.96%
Total runs:
49.7K
Run Growth:
-640
Growth Rate:
-1.29%
Total runs:
41.2K
Run Growth:
41.2K
Growth Rate:
99.97%
Total runs:
35.6K
Run Growth:
26.6K
Growth Rate:
98.51%
Total runs:
31.2K
Run Growth:
-19.4K
Growth Rate:
-62.31%
Total runs:
27.7K
Run Growth:
24.5K
Growth Rate:
88.59%
Total runs:
23.4K
Run Growth:
1.4K
Growth Rate:
5.61%
Total runs:
22.6K
Run Growth:
-54
Growth Rate:
-0.24%
Total runs:
21.2K
Run Growth:
-18.4K
Growth Rate:
-87.09%
Total runs:
17.9K
Run Growth:
-22.8K
Growth Rate:
-127.30%
Total runs:
16.3K
Run Growth:
5.4K
Growth Rate:
27.42%
Total runs:
12.4K
Run Growth:
8.0K
Growth Rate:
69.22%
Total runs:
11.5K
Run Growth:
4.9K
Growth Rate:
42.53%
Total runs:
11.3K
Run Growth:
977
Growth Rate:
8.66%
Total runs:
10.5K
Run Growth:
2.7K
Growth Rate:
25.33%
Total runs:
10.5K
Run Growth:
43
Growth Rate:
0.41%
Total runs:
7.6K
Run Growth:
7.9K
Growth Rate:
65.67%
Total runs:
7.1K
Run Growth:
-19.8K
Growth Rate:
-277.09%
Total runs:
6.9K
Run Growth:
880
Growth Rate:
12.71%
Total runs:
6.5K
Run Growth:
-2.3K
Growth Rate:
-34.31%
Total runs:
6.4K
Run Growth:
3.3K
Growth Rate:
51.01%
Total runs:
6.1K
Run Growth:
-21.5K
Growth Rate:
-354.06%
Total runs:
4.9K
Run Growth:
2.1K
Growth Rate:
45.30%
Total runs:
4.7K
Run Growth:
2.7K
Growth Rate:
56.97%
Total runs:
4.5K
Run Growth:
4.4K
Growth Rate:
97.96%
Total runs:
4.3K
Run Growth:
3.2K
Growth Rate:
74.66%
Total runs:
4.1K
Run Growth:
-888
Growth Rate:
-21.90%
Total runs:
3.7K
Run Growth:
1.6K
Growth Rate:
42.74%
Total runs:
3.3K
Run Growth:
3.1K
Growth Rate:
93.97%
Total runs:
2.4K
Run Growth:
-2.0K
Growth Rate:
-77.45%
Total runs:
2.3K
Run Growth:
1.6K
Growth Rate:
69.43%
Total runs:
2.3K
Run Growth:
2.2K
Growth Rate:
98.27%
Total runs:
2.1K
Run Growth:
2.2K
Growth Rate:
33.30%
Total runs:
2.1K
Run Growth:
578
Growth Rate:
34.24%
Total runs:
2.0K
Run Growth:
-2.5K
Growth Rate:
-123.46%
Total runs:
1.9K
Run Growth:
854
Growth Rate:
44.90%
Total runs:
1.9K
Run Growth:
1.9K
Growth Rate:
99.26%
Total runs:
1.9K
Run Growth:
862
Growth Rate:
45.37%
Total runs:
1.9K
Run Growth:
1.5K
Growth Rate:
82.01%
Total runs:
1.9K
Run Growth:
1.8K
Growth Rate:
97.27%
Total runs:
1.8K
Run Growth:
162
Growth Rate:
9.02%
Total runs:
1.7K
Run Growth:
-160
Growth Rate:
-9.16%
Total runs:
1.7K
Run Growth:
943
Growth Rate:
72.21%
Total runs:
1.6K
Run Growth:
461
Growth Rate:
28.09%
Total runs:
1.6K
Run Growth:
885
Growth Rate:
47.40%
Total runs:
1.5K
Run Growth:
824
Growth Rate:
53.93%
Total runs:
1.4K
Run Growth:
841
Growth Rate:
57.17%
Total runs:
1.0K
Run Growth:
-34
Growth Rate:
-3.30%
Total runs:
946
Run Growth:
-3.2K
Growth Rate:
-239.15%
Total runs:
936
Run Growth:
513
Growth Rate:
54.81%
Total runs:
920
Run Growth:
854
Growth Rate:
92.83%
Total runs:
873
Run Growth:
-148
Growth Rate:
-16.95%
Total runs:
842
Run Growth:
839
Growth Rate:
99.64%
Total runs:
819
Run Growth:
795
Growth Rate:
97.07%
Total runs:
669
Run Growth:
392
Growth Rate:
58.59%
Total runs:
667
Run Growth:
237
Growth Rate:
35.53%
Total runs:
655
Run Growth:
323
Growth Rate:
43.01%
Total runs:
542
Run Growth:
-250
Growth Rate:
-46.13%
Total runs:
522
Run Growth:
-340
Growth Rate:
-65.13%
Total runs:
469
Run Growth:
134
Growth Rate:
29.00%
Total runs:
406
Run Growth:
383
Growth Rate:
94.33%
Total runs:
382
Run Growth:
333
Growth Rate:
87.17%
Total runs:
360
Run Growth:
13
Growth Rate:
4.30%
Total runs:
339
Run Growth:
26
Growth Rate:
7.67%
Total runs:
329
Run Growth:
242
Growth Rate:
73.11%
Total runs:
326
Run Growth:
-125
Growth Rate:
-38.34%
Total runs:
317
Run Growth:
-5.8K
Growth Rate:
-1837.22%
Total runs:
299
Run Growth:
36
Growth Rate:
12.08%
Total runs:
284
Run Growth:
190
Growth Rate:
64.41%
Total runs:
262
Run Growth:
135
Growth Rate:
51.53%
Total runs:
254
Run Growth:
176
Growth Rate:
69.29%
Total runs:
218
Run Growth:
204
Growth Rate:
93.58%
Total runs:
210
Run Growth:
82
Growth Rate:
39.23%
Total runs:
194
Run Growth:
22
Growth Rate:
11.46%
Total runs:
189
Run Growth:
-216
Growth Rate:
-108.00%
Total runs:
177
Run Growth:
162
Growth Rate:
91.53%
Total runs:
176
Run Growth:
-178
Growth Rate:
-101.14%