In this paper, we propose SimLM (Similarity matching with Language Model pre-training), a simple yet effective pre-training method for dense passage retrieval.
It employs a simple bottleneck architecture that learns to compress the passage information into a dense vector through self-supervised pre-training.
We use a replaced language modeling objective, which is inspired by ELECTRA,
to improve the sample efficiency and reduce the mismatch of the input distribution between pre-training and fine-tuning.
SimLM only requires access to unlabeled corpus, and is more broadly applicable when there are no labeled data or queries.
We conduct experiments on several large-scale passage retrieval datasets, and show substantial improvements over strong baselines under various settings.
Remarkably, SimLM even outperforms multi-vector approaches such as ColBERTv2 which incurs significantly more storage cost.
Results on MS-MARCO passage ranking task
Model
dev MRR@10
dev R@50
dev R@1k
TREC DL 2019 nDCG@10
TREC DL 2020 nDCG@10
SimLM (this model)
43.8
89.2
98.6
74.6
72.7
Usage
Since we use a listwise loss to train the re-ranker,
the relevance score is not bounded to a specific numerical range.
Higher scores mean more relevant between the given query and passage.
Get relevance score from our re-ranker:
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer, BatchEncoding, PreTrainedTokenizerFast
from transformers.modeling_outputs import SequenceClassifierOutput
defencode(tokenizer: PreTrainedTokenizerFast, query: str, passage: str, title: str = '-') -> BatchEncoding:
return tokenizer(query,
text_pair='{}: {}'.format(title, passage),
max_length=192,
padding=True,
truncation=True,
return_tensors='pt')
tokenizer = AutoTokenizer.from_pretrained('intfloat/simlm-msmarco-reranker')
model = AutoModelForSequenceClassification.from_pretrained('intfloat/simlm-msmarco-reranker')
model.eval()
with torch.no_grad():
batch_dict = encode(tokenizer, 'how long is super bowl game', 'The Super Bowl is typically four hours long. The game itself takes about three and a half hours, with a 30 minute halftime show built in.')
outputs: SequenceClassifierOutput = model(**batch_dict, return_dict=True)
print(outputs.logits[0])
batch_dict = encode(tokenizer, 'how long is super bowl game', 'The cost of a Super Bowl commercial runs about $5 million for 30 seconds of airtime. But the benefits that the spot can bring to a brand can help to justify the cost.')
outputs: SequenceClassifierOutput = model(**batch_dict, return_dict=True)
print(outputs.logits[0])
Citation
@article{Wang2022SimLMPW,
title={SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval},
author={Liang Wang and Nan Yang and Xiaolong Huang and Binxing Jiao and Linjun Yang and Daxin Jiang and Rangan Majumder and Furu Wei},
journal={ArXiv},
year={2022},
volume={abs/2207.02578}
}
Runs of intfloat simlm-msmarco-reranker on huggingface.co
3.9K
Total runs
0
24-hour runs
2
3-day runs
-235
7-day runs
597
30-day runs
More Information About simlm-msmarco-reranker huggingface.co Model
simlm-msmarco-reranker huggingface.co is an AI model on huggingface.co that provides simlm-msmarco-reranker's model effect (), which can be used instantly with this intfloat simlm-msmarco-reranker model. huggingface.co supports a free trial of the simlm-msmarco-reranker model, and also provides paid use of the simlm-msmarco-reranker. Support call simlm-msmarco-reranker model through api, including Node.js, Python, http.
simlm-msmarco-reranker huggingface.co is an online trial and call api platform, which integrates simlm-msmarco-reranker's modeling effects, including api services, and provides a free online trial of simlm-msmarco-reranker, you can try simlm-msmarco-reranker online for free by clicking the link below.
intfloat simlm-msmarco-reranker online free url in huggingface.co:
simlm-msmarco-reranker is an open source model from GitHub that offers a free installation service, and any user can find simlm-msmarco-reranker on GitHub to install. At the same time, huggingface.co provides the effect of simlm-msmarco-reranker install, users can directly use simlm-msmarco-reranker installed effect in huggingface.co for debugging and trial. It also supports api for free installation.
simlm-msmarco-reranker install url in huggingface.co: