Total runs:
20.3K
Run Growth:
-1.2K
Growth Rate:
-5.92%
castorini / rank_zephyr_7b_v1_full
Model's Last Updated: January 19 2024
text-generation
Introduction of rank_zephyr_7b_v1_full
Model Details of rank_zephyr_7b_v1_full
Model Card for RankZephyr 7B V1 - Full
RankZephyr is a series of language models trained to act as helpful reranking assistants built on the Zephyr-7B-β model. RankZephyr Base is the model that follows single-stage fine-tuning on the RankGPT-3.5 model, while RankZephyr Full is the model that is further fine-tuned on RankGPT-4 reorderings of OpenAI's Ada2 orderings for 5K queries.
Model description
- Model type: A 7B parameter GPT-like model initially fine-tuned on a mix of publicly available, synthetic datasets, followed by task-specific listwise reranking data.
- Language(s) (NLP): Primarily English
- License: MIT
- Fine-tuned from model: HuggingFaceH4/zephyr-7b-beta
Model Sources
- Repository: https://github.com/castorini/rank_llm
- Paper: https://arxiv.org/abs/2312.02724
Effectiveness
At the time of release, RankZephyr-7B-Full is the state-of-the-art open-source reranking model on various datasets like DL19/20/21/22 and TREC-COVID and TREC-News.
With the MS MARCO v1 collection:
| Model | Size | First Stage | DL19 | DL20 |
|---|---|---|---|---|
| RankZephyr-7b-v1-full-rho 🪁 | 7B | SPLADE++ ED | 0.7855 | 0.8255 |
| RankZephyr-7b-v1-full 🪁 | 7B | SPLADE++ ED | 0.7803 | 0.8211 |
| RankGPT-4 (PSC) | - | SPLADE++ ED | 0.7601 | 0.7514 |
| RankGPT-4 | - | SPLADE++ ED | 0.7464 | 0.7076 |
| RankZephyr-7b-v1-base 🪁 | 7B | SPLADE++ ED | 0.7341 | 0.7213 |
| RankGPT-3.5 | - | SPLADE++ ED | 0.7504 | 0.7120 |
More details can be found in the paper.
Intended uses & limitations
The model is to be used in conjunction with the
RankLLM repository
. While
rank-llm
exists as a PyPI package, we are currently in the early stages of development and encourage users to directly check install from source.
The original Zephyr model is trained for chat. In our case, RankZephyr is fine-tuned to act as a listwise reranking agent. You provide it with a query and documents and get back a reordered list of document identifiers.
Bias, Risks, and Limitations
The following is an excerpt from the Zephyr-7B-β model card :
Zephyr-7B-β has not been aligned to human preferences for safety within the RLHF phase or deployed with in-the-loop filtering of responses like ChatGPT, so the model can produce problematic outputs (especially when prompted to do so). It is also unknown what the size and composition of the corpus was used to train the base model (
mistralai/Mistral-7B-v0.1), however it is likely to have included a mix of Web data and technical sources like books and code. See the Falcon 180B model card for an example of this.
Our model is trained specifically on monolingual English data, effectiveness on multilingual sets is not guaranteed.
Citation
If you find RankZephyr is useful in your work, please cite the following paper:
@ARTICLE{pradeep2023rankzephyr,
title = {{RankZephyr}: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!},
author = {Ronak Pradeep and Sahel Sharifymoghaddam and Jimmy Lin},
year = {2023},
journal = {arXiv:2312.02724}
}
Runs of castorini rank_zephyr_7b_v1_full on huggingface.co
1.9K
Total runs
97
24-hour runs
150
3-day runs
161
7-day runs
-282
30-day runs
More Information About rank_zephyr_7b_v1_full huggingface.co Model
Url of rank_zephyr_7b_v1_full
rank_zephyr_7b_v1_full huggingface.co Url
Provider of rank_zephyr_7b_v1_full huggingface.co
Other API from castorini
Updated: November 24 2021
Total runs:
7.2K
Run Growth:
5.7K
Growth Rate:
79.04%
Total runs:
2.8K
Run Growth:
-10.5K
Growth Rate:
-374.39%
Total runs:
2.8K
Run Growth:
-3.5K
Growth Rate:
-119.76%
Total runs:
2.5K
Run Growth:
-620
Growth Rate:
-25.07%
Total runs:
2.0K
Run Growth:
425
Growth Rate:
21.28%
Total runs:
1.4K
Run Growth:
-43
Growth Rate:
-3.06%
Total runs:
1.2K
Run Growth:
317
Growth Rate:
26.53%
Total runs:
741
Run Growth:
628
Growth Rate:
84.41%
Total runs:
739
Run Growth:
212
Growth Rate:
29.65%
Total runs:
670
Run Growth:
92
Growth Rate:
13.73%
Total runs:
528
Run Growth:
268
Growth Rate:
50.66%
Total runs:
498
Run Growth:
347
Growth Rate:
69.40%
Total runs:
477
Run Growth:
339
Growth Rate:
71.07%
Total runs:
441
Run Growth:
-22
Growth Rate:
-4.99%
Total runs:
437
Run Growth:
-794
Growth Rate:
-200.51%
Total runs:
422
Run Growth:
403
Growth Rate:
95.50%
Total runs:
285
Run Growth:
-192
Growth Rate:
-66.90%
Total runs:
264
Run Growth:
-171
Growth Rate:
-66.28%
Total runs:
262
Run Growth:
-174
Growth Rate:
-67.97%
Total runs:
257
Run Growth:
247
Growth Rate:
96.48%
Total runs:
235
Run Growth:
-6.4K
Growth Rate:
-2685.36%
Total runs:
233
Run Growth:
217
Growth Rate:
97.31%
Total runs:
211
Run Growth:
98
Growth Rate:
46.45%
Total runs:
178
Run Growth:
154
Growth Rate:
86.52%
Total runs:
168
Run Growth:
26
Growth Rate:
15.48%
Total runs:
166
Run Growth:
-15
Growth Rate:
-8.93%
Total runs:
164
Run Growth:
82
Growth Rate:
49.40%
Total runs:
161
Run Growth:
154
Growth Rate:
93.90%
Total runs:
159
Run Growth:
113
Growth Rate:
71.07%
Total runs:
155
Run Growth:
125
Growth Rate:
80.65%
Total runs:
142
Run Growth:
115
Growth Rate:
77.70%
Total runs:
137
Run Growth:
112
Growth Rate:
81.75%
Total runs:
137
Run Growth:
-407
Growth Rate:
-297.08%
Total runs:
126
Run Growth:
-78
Growth Rate:
-60.94%
Total runs:
124
Run Growth:
108
Growth Rate:
86.40%
Total runs:
123
Run Growth:
95
Growth Rate:
77.24%
Total runs:
119
Run Growth:
-183
Growth Rate:
-151.24%
Total runs:
118
Run Growth:
62
Growth Rate:
51.67%
Total runs:
117
Run Growth:
118
Growth Rate:
99.16%
Total runs:
114
Run Growth:
114
Growth Rate:
97.44%
Total runs:
111
Run Growth:
111
Growth Rate:
98.23%
Total runs:
108
Run Growth:
44
Growth Rate:
40.00%
Total runs:
108
Run Growth:
73
Growth Rate:
66.36%
Total runs:
107
Run Growth:
99
Growth Rate:
92.52%
Total runs:
101
Run Growth:
80
Growth Rate:
76.92%
Total runs:
97
Run Growth:
93
Growth Rate:
95.88%
Total runs:
95
Run Growth:
-108
Growth Rate:
-111.34%
Total runs:
88
Run Growth:
77
Growth Rate:
87.50%
Total runs:
86
Run Growth:
-64
Growth Rate:
-59.81%
Total runs:
41
Run Growth:
37
Growth Rate:
90.24%
Total runs:
41
Run Growth:
-69
Growth Rate:
-168.29%
Total runs:
40
Run Growth:
27
Growth Rate:
67.50%
Total runs:
39
Run Growth:
8
Growth Rate:
20.51%
Total runs:
37
Run Growth:
5
Growth Rate:
13.51%
Total runs:
37
Run Growth:
6
Growth Rate:
16.22%
Total runs:
37
Run Growth:
5
Growth Rate:
13.51%
Total runs:
37
Run Growth:
31
Growth Rate:
83.78%
Total runs:
37
Run Growth:
6
Growth Rate:
16.22%
Total runs:
37
Run Growth:
6
Growth Rate:
16.22%
Total runs:
36
Run Growth:
8
Growth Rate:
22.22%
Total runs:
35
Run Growth:
-265
Growth Rate:
-757.14%
Total runs:
29
Run Growth:
-3
Growth Rate:
-10.34%
Total runs:
29
Run Growth:
26
Growth Rate:
89.66%
Total runs:
29
Run Growth:
18
Growth Rate:
62.07%
Total runs:
28
Run Growth:
21
Growth Rate:
75.00%
Total runs:
24
Run Growth:
-93
Growth Rate:
-387.50%
Total runs:
22
Run Growth:
23
Growth Rate:
92.00%
Total runs:
22
Run Growth:
15
Growth Rate:
65.22%
Total runs:
21
Run Growth:
-85
Growth Rate:
-404.76%
Total runs:
20
Run Growth:
13
Growth Rate:
65.00%
Total runs:
20
Run Growth:
-120
Growth Rate:
-600.00%
Total runs:
20
Run Growth:
-66
Growth Rate:
-330.00%
Total runs:
19
Run Growth:
-120
Growth Rate:
-631.58%
Total runs:
18
Run Growth:
-89
Growth Rate:
-494.44%
Total runs:
18
Run Growth:
-93
Growth Rate:
-516.67%
Total runs:
16
Run Growth:
-92
Growth Rate:
-575.00%
Total runs:
16
Run Growth:
11
Growth Rate:
68.75%
Total runs:
15
Run Growth:
-2
Growth Rate:
-13.33%
Total runs:
14
Run Growth:
-94
Growth Rate:
-671.43%
Total runs:
14
Run Growth:
-95
Growth Rate:
-678.57%
Total runs:
13
Run Growth:
-94
Growth Rate:
-723.08%
Total runs:
13
Run Growth:
-92
Growth Rate:
-707.69%
Total runs:
13
Run Growth:
-96
Growth Rate:
-738.46%
Total runs:
13
Run Growth:
-94
Growth Rate:
-723.08%
Total runs:
12
Run Growth:
-98
Growth Rate:
-816.67%
Total runs:
12
Run Growth:
-72
Growth Rate:
-600.00%
Total runs:
12
Run Growth:
9
Growth Rate:
64.29%
Total runs:
11
Run Growth:
-134
Growth Rate:
-1218.18%