BioMed-RoBERTa-base is a language model based on the RoBERTa-base (Liu et. al, 2019) architecture. We adapt RoBERTa-base to 2.68 million scientific papers from the
Semantic Scholar
corpus via continued pretraining. This amounts to 7.55B tokens and 47GB of data. We use the full text of the papers in training, not just abstracts.
Specific details of the adaptive pretraining procedure can be found in Gururangan et. al, 2020.
Evaluation
BioMed-RoBERTa achieves competitive performance to state of the art models on a number of NLP tasks in the biomedical domain (numbers are mean (standard deviation) over 3+ random seeds)
Task
Task Type
RoBERTa-base
BioMed-RoBERTa-base
RCT-180K
Text Classification
86.4 (0.3)
86.9 (0.2)
ChemProt
Relation Extraction
81.1 (1.1)
83.0 (0.7)
JNLPBA
NER
74.3 (0.2)
75.2 (0.1)
BC5CDR
NER
85.6 (0.1)
87.8 (0.1)
NCBI-Disease
NER
86.6 (0.3)
87.1 (0.8)
More evaluations TBD.
Citation
If using this model, please cite the following paper:
@inproceedings{domains,
author = {Suchin Gururangan and Ana Marasović and Swabha Swayamdipta and Kyle Lo and Iz Beltagy and Doug Downey and Noah A. Smith},
title = {Don't Stop Pretraining: Adapt Language Models to Domains and Tasks},
year = {2020},
booktitle = {Proceedings of ACL},
}
Runs of allenai biomed_roberta_base on huggingface.co
49.7K
Total runs
0
24-hour runs
-516
3-day runs
-689
7-day runs
-640
30-day runs
More Information About biomed_roberta_base huggingface.co Model
biomed_roberta_base huggingface.co
biomed_roberta_base huggingface.co is an AI model on huggingface.co that provides biomed_roberta_base's model effect (), which can be used instantly with this allenai biomed_roberta_base model. huggingface.co supports a free trial of the biomed_roberta_base model, and also provides paid use of the biomed_roberta_base. Support call biomed_roberta_base model through api, including Node.js, Python, http.
biomed_roberta_base huggingface.co is an online trial and call api platform, which integrates biomed_roberta_base's modeling effects, including api services, and provides a free online trial of biomed_roberta_base, you can try biomed_roberta_base online for free by clicking the link below.
allenai biomed_roberta_base online free url in huggingface.co:
biomed_roberta_base is an open source model from GitHub that offers a free installation service, and any user can find biomed_roberta_base on GitHub to install. At the same time, huggingface.co provides the effect of biomed_roberta_base install, users can directly use biomed_roberta_base installed effect in huggingface.co for debugging and trial. It also supports api for free installation.
biomed_roberta_base install url in huggingface.co: