In the paper, Gao & Callan claim a MS MARCO-Dev score of 38.2 (MRR@10). This is achieved by changing the benchmark: The orginal MS MARCO dataset just provides queries and text passages, from which you must retrieve the relevant passages for a given query.
In their
code
, they combine the passages with the document titles from MS MARCO document task, i.e. they train and evaluate their model with additional information from a different benchmark. In the above table, the score of 35.41 (MRR@10) is on the MS MARCO Passages benchmark as it is proposed, without having the document titles.
They further trained their model with the document titles, which creates an information leackage: The document titles were re-constructed by the MS MARCO organizers at a later stage for the MS MARCO document benchmark. It was not possible to reconstruct all document titles for all passages. However, the distribution of having a title is not equal for relevant and non-relevant passages: 71.9% of the relevant passages have a document title, while only 64.4% of the non-relevant passages have a title. Hence, the model can learn that, as soon as there is a document title, the probability is higher that this passage is annotated as relevant. It will not make the decision based on the passage content, but by the artifact if there is a title or not.
The information leackage and the change of the benchmark likely leads to the inflated scores reported in the paper.
from sentence_transformers import SentenceTransformer, util
query = "How many people live in London?"
docs = ["Around 9 Million people live in London", "London is known for its financial district"]
#Load the model
model = SentenceTransformer('sentence-transformers/msmarco-bert-co-condensor')
#Encode query and documents
query_emb = model.encode(query)
doc_emb = model.encode(docs)
#Compute dot score between query and all document embeddings
scores = util.dot_score(query_emb, doc_emb)[0].cpu().tolist()
#Combine docs & scores
doc_score_pairs = list(zip(docs, scores))
#Sort by decreasing score
doc_score_pairs = sorted(doc_score_pairs, key=lambda x: x[1], reverse=True)
#Output passages & scoresfor doc, score in doc_score_pairs:
print(score, doc)
Usage (HuggingFace Transformers)
Without
sentence-transformers
, you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
from transformers import AutoTokenizer, AutoModel
import torch
#CLS Pooling - Take output from first tokendefcls_pooling(model_output):
return model_output.last_hidden_state[:,0]
#Encode textdefencode(texts):
# Tokenize sentences
encoded_input = tokenizer(texts, padding=True, truncation=True, return_tensors='pt')
# Compute token embeddingswith torch.no_grad():
model_output = model(**encoded_input, return_dict=True)
# Perform pooling
embeddings = cls_pooling(model_output)
return embeddings
# Sentences we want sentence embeddings for
query = "How many people live in London?"
docs = ["Around 9 Million people live in London", "London is known for its financial district"]
# Load model from HuggingFace Hub
tokenizer = AutoTokenizer.from_pretrained("sentence-transformers/msmarco-bert-co-condensor")
model = AutoModel.from_pretrained("sentence-transformers/msmarco-bert-co-condensor")
#Encode query and docs
query_emb = encode(query)
doc_emb = encode(docs)
#Compute dot score between query and all document embeddings
scores = torch.mm(query_emb, doc_emb.transpose(0, 1))[0].cpu().tolist()
#Combine docs & scores
doc_score_pairs = list(zip(docs, scores))
#Sort by decreasing score
doc_score_pairs = sorted(doc_score_pairs, key=lambda x: x[1], reverse=True)
#Output passages & scoresfor doc, score in doc_score_pairs:
print(score, doc)
Evaluation Results
For an automated evaluation of this model, see the
Sentence Embeddings Benchmark
:
https://seb.sbert.net
msmarco-bert-co-condensor huggingface.co is an AI model on huggingface.co that provides msmarco-bert-co-condensor's model effect (), which can be used instantly with this sentence-transformers msmarco-bert-co-condensor model. huggingface.co supports a free trial of the msmarco-bert-co-condensor model, and also provides paid use of the msmarco-bert-co-condensor. Support call msmarco-bert-co-condensor model through api, including Node.js, Python, http.
msmarco-bert-co-condensor huggingface.co is an online trial and call api platform, which integrates msmarco-bert-co-condensor's modeling effects, including api services, and provides a free online trial of msmarco-bert-co-condensor, you can try msmarco-bert-co-condensor online for free by clicking the link below.
sentence-transformers msmarco-bert-co-condensor online free url in huggingface.co:
msmarco-bert-co-condensor is an open source model from GitHub that offers a free installation service, and any user can find msmarco-bert-co-condensor on GitHub to install. At the same time, huggingface.co provides the effect of msmarco-bert-co-condensor install, users can directly use msmarco-bert-co-condensor installed effect in huggingface.co for debugging and trial. It also supports api for free installation.
msmarco-bert-co-condensor install url in huggingface.co: