intfloat / simlm-base-msmarco-finetuned

huggingface.co
Total runs: 7.0K
24-hour runs: 0
7-day runs: -10
30-day runs: 5.6K
Model's Last Updated: 2023年5月22日
feature-extraction

Introduction of simlm-base-msmarco-finetuned

Model Details of simlm-base-msmarco-finetuned

SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval

paper available at https://arxiv.org/pdf/2207.02578

code available at https://github.com/microsoft/unilm/tree/master/simlm

Paper abstract

In this paper, we propose SimLM (Similarity matching with Language Model pre-training), a simple yet effective pre-training method for dense passage retrieval. It employs a simple bottleneck architecture that learns to compress the passage information into a dense vector through self-supervised pre-training. We use a replaced language modeling objective, which is inspired by ELECTRA, to improve the sample efficiency and reduce the mismatch of the input distribution between pre-training and fine-tuning. SimLM only requires access to unlabeled corpus, and is more broadly applicable when there are no labeled data or queries. We conduct experiments on several large-scale passage retrieval datasets, and show substantial improvements over strong baselines under various settings. Remarkably, SimLM even outperforms multi-vector approaches such as ColBERTv2 which incurs significantly more storage cost.

Results on MS-MARCO passage ranking task
Model dev MRR@10 dev R@50 dev R@1k TREC DL 2019 nDCG@10 TREC DL 2020 nDCG@10
RocketQAv2 38.8 86.2 98.1 - -
coCondenser 38.2 86.5 98.4 71.7 68.4
ColBERTv2 39.7 86.8 98.4 - -
SimLM (this model) 41.1 87.8 98.7 71.4 69.7
Usage

Get embeddings from our fine-tuned model:

import torch
from transformers import AutoModel, AutoTokenizer, BatchEncoding, PreTrainedTokenizerFast
from transformers.modeling_outputs import BaseModelOutput

def l2_normalize(x: torch.Tensor):
    return torch.nn.functional.normalize(x, p=2, dim=-1)

def encode_query(tokenizer: PreTrainedTokenizerFast, query: str) -> BatchEncoding:
    return tokenizer(query,
                     max_length=32,
                     padding=True,
                     truncation=True,
                     return_tensors='pt')

def encode_passage(tokenizer: PreTrainedTokenizerFast, passage: str, title: str = '-') -> BatchEncoding:
    return tokenizer(title,
                     text_pair=passage,
                     max_length=144,
                     padding=True,
                     truncation=True,
                     return_tensors='pt')

tokenizer = AutoTokenizer.from_pretrained('intfloat/simlm-base-msmarco-finetuned')
model = AutoModel.from_pretrained('intfloat/simlm-base-msmarco-finetuned')
model.eval()

with torch.no_grad():
    query_batch_dict = encode_query(tokenizer, 'what is qa')
    outputs: BaseModelOutput = model(**query_batch_dict, return_dict=True)
    query_embedding = l2_normalize(outputs.last_hidden_state[0, 0, :])

    psg1 = 'Quality assurance (QA) is a process-centered approach to ensuring that a company or organization is providing the best possible products or services. It is related to quality control, which focuses on the end result, such as testing a sample of items from a batch after production.'
    psg1_batch_dict = encode_passage(tokenizer, psg1)
    outputs: BaseModelOutput = model(**psg1_batch_dict, return_dict=True)
    psg1_embedding = l2_normalize(outputs.last_hidden_state[0, 0, :])

    psg2 = 'The Super Bowl is typically four hours long. The game itself takes about three and a half hours, with a 30 minute halftime show built in.'
    psg2_batch_dict = encode_passage(tokenizer, psg2)
    outputs: BaseModelOutput = model(**psg2_batch_dict, return_dict=True)
    psg2_embedding = l2_normalize(outputs.last_hidden_state[0, 0, :])

    # Higher cosine similarity means they are more relevant
    print(query_embedding.dot(psg1_embedding), query_embedding.dot(psg2_embedding))
Citation
@article{Wang2022SimLMPW,
  title={SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval},
  author={Liang Wang and Nan Yang and Xiaolong Huang and Binxing Jiao and Linjun Yang and Daxin Jiang and Rangan Majumder and Furu Wei},
  journal={ArXiv},
  year={2022},
  volume={abs/2207.02578}
}

Runs of intfloat simlm-base-msmarco-finetuned on huggingface.co

7.0K
Total runs
0
24-hour runs
-44
3-day runs
-10
7-day runs
5.6K
30-day runs

More Information About simlm-base-msmarco-finetuned huggingface.co Model

More simlm-base-msmarco-finetuned license Visit here:

https://choosealicense.com/licenses/mit

simlm-base-msmarco-finetuned huggingface.co

simlm-base-msmarco-finetuned huggingface.co is an AI model on huggingface.co that provides simlm-base-msmarco-finetuned's model effect (), which can be used instantly with this intfloat simlm-base-msmarco-finetuned model. huggingface.co supports a free trial of the simlm-base-msmarco-finetuned model, and also provides paid use of the simlm-base-msmarco-finetuned. Support call simlm-base-msmarco-finetuned model through api, including Node.js, Python, http.

simlm-base-msmarco-finetuned huggingface.co Url

https://huggingface.co/intfloat/simlm-base-msmarco-finetuned

intfloat simlm-base-msmarco-finetuned online free

simlm-base-msmarco-finetuned huggingface.co is an online trial and call api platform, which integrates simlm-base-msmarco-finetuned's modeling effects, including api services, and provides a free online trial of simlm-base-msmarco-finetuned, you can try simlm-base-msmarco-finetuned online for free by clicking the link below.

intfloat simlm-base-msmarco-finetuned online free url in huggingface.co:

https://huggingface.co/intfloat/simlm-base-msmarco-finetuned

simlm-base-msmarco-finetuned install

simlm-base-msmarco-finetuned is an open source model from GitHub that offers a free installation service, and any user can find simlm-base-msmarco-finetuned on GitHub to install. At the same time, huggingface.co provides the effect of simlm-base-msmarco-finetuned install, users can directly use simlm-base-msmarco-finetuned installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

simlm-base-msmarco-finetuned install url in huggingface.co:

https://huggingface.co/intfloat/simlm-base-msmarco-finetuned

Url of simlm-base-msmarco-finetuned

simlm-base-msmarco-finetuned huggingface.co Url

Provider of simlm-base-msmarco-finetuned huggingface.co

intfloat
ORGANIZATIONS

Other API from intfloat

huggingface.co

Total runs: 1.1M
Run Growth: 76.7K
Growth Rate: 7.02%
Updated: 2023年8月7日
huggingface.co

Total runs: 490.9K
Run Growth: 166.0K
Growth Rate: 33.76%
Updated: 2023年9月27日
huggingface.co

Total runs: 333.7K
Run Growth: 219.4K
Growth Rate: 65.74%
Updated: 2023年8月7日
huggingface.co

Total runs: 129.2K
Run Growth: 20.8K
Growth Rate: 16.43%
Updated: 2023年8月16日
huggingface.co

Total runs: 57.3K
Run Growth: 26.9K
Growth Rate: 46.97%
Updated: 2023年8月7日
huggingface.co

Total runs: 17.1K
Run Growth: -11.6K
Growth Rate: -67.89%
Updated: 2023年8月7日