allenai / specter2_aug2023refresh_adhoc_query

SPECTER2

SPECTER2 is a family of models that succeeds SPECTER and is capable of generating task specific embeddings for scientific tasks when paired with adapters . Given the combination of title and abstract of a scientific paper or a short texual query, the model can be used to generate effective embeddings to be used in downstream applications.

Note:For general embedding purposes, please use allenai/specter2 .

To get the best performance on a downstream task type please load the associated adapter () with the base model as in the example below.

Dec 2023 Update:

Model usage updated to be compatible with latest versions of transformers and adapters (newly released update to adapter-transformers) libraries.

******Update******

This update introduces a new set of SPECTER 2.0 models with the base transformer encoder pre-trained on an extended citation dataset containing more recent papers. For benchmarking purposes please use the existing SPECTER 2.0 models w/o the aug2023refresh suffix viz. allenai/specter2_base .

Adapter allenai/specter2_aug2023refresh_adhoc_query for allenai/specter2_aug2023refresh_base

An adapter for the None model that was trained on the allenai/scirepeval dataset.

This adapter was created for usage with the adapters library.

Usage

First, install adapters :

pip install -U adapters

Note: adapters is built as an add-on to transformers that acts as a drop-in replacement with adapter support. More

Now, the adapter can be loaded and activated like this:

from adapters import AutoAdapterModel

model = AutoAdapterModel.from_pretrained("allenai/specter2_aug2023refresh_base")
adapter_name = model.load_adapter("allenai/specter2_aug2023refresh_adhoc_query", source="hf", set_active=True)

Model Details

Model Description

SPECTER 2.0 has been trained on over 6M triplets of scientific paper citations, which are available here . Post that it is trained with additionally attached task format specific adapter modules on all the SciRepEval training tasks.

Task Formats trained on:

• Classification
• Regression
• Proximity
• Adhoc Search

This is the adhoc search query specific adapter. For tasks where papers have to retrieved for a short textual query, use this adapter to encode the query and allenai/specter2_proximity to encode the candidates.

It builds on the work done in SciRepEval: A Multi-Format Benchmark for Scientific Document Representations and we evaluate the trained model on this benchmark as well.

• Developed by: Amanpreet Singh, Mike D'Arcy, Arman Cohan, Doug Downey, Sergey Feldman
• Shared by : Allen AI
• Model type: bert-base-uncased + adapters
• License: Apache 2.0
• Finetuned from model: allenai/scibert .

Model Sources

• Repository: https://github.com/allenai/SPECTER2_0
• Paper: https://api.semanticscholar.org/CorpusID:254018137
• Demo: Usage

Uses

Direct Use

*Proximity model should suffice for downstream task types not mentioned above

from transformers import AutoTokenizer
from adapters import AutoAdapterModel
from sklearn.metrics.pairwise import euclidean_distances

def embed_input(text_batch: List[str]):
  # preprocess the input
  inputs = self.tokenizer(text_batch, padding=True, truncation=True,
                                   return_tensors="pt", return_token_type_ids=False, max_length=512)
  output = model(**inputs)
  # take the first token in the batch as the embedding
  embeddings = output.last_hidden_state[:, 0, :]
  return embeddings

# load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained('allenai/specter2_aug2023refresh_base_base')

#load base model
model = AutoAdapterModel.from_pretrained('allenai/specter2_aug2023refresh_base_aug2023refresh_base_base')

#load the query adapter, provide an identifier for the adapter in load_as argument and activate it
model.load_adapter("allenai/specter2_aug2023refresh_base_adhoc_query", source="hf", load_as="specter2_adhoc_query", set_active=True)
query = ["Bidirectional transformers"]
query_embedding = embed_input(query)

#load the proximity adapter, provide an identifier for the adapter in load_as argument and activate it
model.load_adapter("allenai/specter2_aug2023refresh_base", source="hf", load_as="specter2_proximity", set_active=True)
papers = [{'title': 'BERT', 'abstract': 'We introduce a new language representation model called BERT'},
          {'title': 'Attention is all you need', 'abstract': ' The dominant sequence transduction models are based on complex recurrent or convolutional neural networks'}]
# concatenate title and abstract
text_papers_batch = [d['title'] + tokenizer.sep_token + (d.get('abstract') or '') for d in papers]
paper_embeddings = embed_input(text_papers_batch)

#Calculate L2 distance between query and papers
l2_distance = euclidean_distances(papers, query).flatten()

Downstream Use

For evaluation and downstream usage, please refer to https://github.com/allenai/scirepeval/blob/main/evaluation/INFERENCE.md .

Training Details

Training Data

The base model is trained on citation links between papers and the adapters are trained on 8 large scale tasks across the four formats. All the data is a part of SciRepEval benchmark and is available here .

The citation link are triplets in the form

{"query": {"title": ..., "abstract": ...}, "pos": {"title": ..., "abstract": ...}, "neg": {"title": ..., "abstract": ...}}

consisting of a query paper, a positive citation and a negative which can be from the same/different field of study as the query or citation of a citation.

Training Procedure

Please refer to the SPECTER paper .

Training Hyperparameters

The model is trained in two stages using SciRepEval :

• Base Model: First a base model is trained on the above citation triplets.

batch size = 1024, max input length = 512, learning rate = 2e-5, epochs = 2 warmup steps = 10% fp16

• Adapters: Thereafter, task format specific adapters are trained on the SciRepEval training tasks, where 600K triplets are sampled from above and added to the training data as well.

batch size = 256, max input length = 512, learning rate = 1e-4, epochs = 6 warmup = 1000 steps fp16

Evaluation

We evaluate the model on SciRepEval , a large scale eval benchmark for scientific embedding tasks which which has [SciDocs] as a subset. We also evaluate and establish a new SoTA on MDCR , a large scale citation recommendation benchmark.

Please cite the following works if you end up using SPECTER 2.0:

SPECTER paper :

@inproceedings{specter2020cohan,
  title={{SPECTER: Document-level Representation Learning using Citation-informed Transformers}},
  author={Arman Cohan and Sergey Feldman and Iz Beltagy and Doug Downey and Daniel S. Weld},
  booktitle={ACL},
  year={2020}
}

SciRepEval paper

@inproceedings{Singh2022SciRepEvalAM,
  title={SciRepEval: A Multi-Format Benchmark for Scientific Document Representations},
  author={Amanpreet Singh and Mike D'Arcy and Arman Cohan and Doug Downey and Sergey Feldman},
  booktitle={Conference on Empirical Methods in Natural Language Processing},
  year={2022},
  url={https://api.semanticscholar.org/CorpusID:254018137}
}