Total runs:
1.9M
Run Growth:
74.2K
Growth Rate:
3.91%
deepset / tinyroberta-6l-768d
Model's Last Updated: September 26 2024
question-answering
Introduction of tinyroberta-6l-768d
Model Details of tinyroberta-6l-768d
tinyroberta-squad2
Overview
Language model:
tinyroberta-squad2
Language:
English
Training data:
The PILE
Code:
Infrastructure
: 4x Tesla v100
Hyperparameters
batch_size = 96
n_epochs = 4
base_LM_model = "deepset/tinyroberta-squad2-step1"
max_seq_len = 384
learning_rate = 1e-4
lr_schedule = LinearWarmup
warmup_proportion = 0.2
teacher = "deepset/roberta-base"
Distillation
This model was distilled using the TinyBERT approach described in this paper and implemented in haystack . We have performed intermediate layer distillation with roberta-base as the teacher which resulted in deepset/tinyroberta-6l-768d . This model has not been distilled for any specific task. If you are interested in using distillation to improve its performance on a downstream task, you can take advantage of haystack's new distillation functionality . You can also check out deepset/tinyroberta-squad2 for a model that is already distilled on an extractive QA downstream task.
Usage
In Transformers
from transformers import AutoModelForQuestionAnswering, AutoTokenizer, pipeline
model_name = "deepset/tinyroberta-squad2"
model = AutoModelForQuestionAnswering.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
In FARM
from farm.modeling.adaptive_model import AdaptiveModel
from farm.modeling.tokenization import Tokenizer
from farm.infer import Inferencer
model_name = "deepset/tinyroberta-squad2"
model = AdaptiveModel.convert_from_transformers(model_name, device="cpu", task_type="question_answering")
tokenizer = Tokenizer.load(model_name)
In haystack
For doing QA at scale (i.e. many docs instead of single paragraph), you can load the model also in haystack :
reader = FARMReader(model_name_or_path="deepset/roberta-base-squad2")
# or
reader = TransformersReader(model_name_or_path="deepset/roberta-base-squad2",tokenizer="deepset/roberta-base-squad2")
Authors
Branden Chan:
branden.chan [at] deepset.ai
Timo Möller:
timo.moeller [at] deepset.ai
Malte Pietsch:
malte.pietsch [at] deepset.ai
Tanay Soni:
tanay.soni [at] deepset.ai
Michel Bartels:
michel.bartels [at] deepset.ai
About us
Our focus: Industry specific language models & large scale QA systems.
Some of our work:
- German BERT (aka "bert-base-german-cased")
- GermanQuAD and GermanDPR datasets and models (aka "gelectra-base-germanquad", "gbert-base-germandpr")
- FARM
- Haystack
Get in touch: Twitter | LinkedIn | Slack | GitHub Discussions | Website
By the way: we're hiring!
Runs of deepset tinyroberta-6l-768d on huggingface.co
65
Total runs
0
24-hour runs
0
3-day runs
21
7-day runs
32
30-day runs
More Information About tinyroberta-6l-768d huggingface.co Model
Url of tinyroberta-6l-768d
tinyroberta-6l-768d huggingface.co Url
Provider of tinyroberta-6l-768d huggingface.co
Other API from deepset
Updated: September 24 2024
Total runs:
222.5K
Run Growth:
11.0K
Growth Rate:
4.93%
Total runs:
194.3K
Run Growth:
-1.1M
Growth Rate:
-465.42%
Total runs:
93.7K
Run Growth:
6.6K
Growth Rate:
7.21%
Total runs:
44.5K
Run Growth:
28.8K
Growth Rate:
68.52%
Total runs:
29.9K
Run Growth:
-290
Growth Rate:
-0.97%
Total runs:
29.5K
Run Growth:
14.1K
Growth Rate:
50.91%
Total runs:
27.3K
Run Growth:
5.4K
Growth Rate:
19.96%
Total runs:
27.0K
Run Growth:
-77.0K
Growth Rate:
-290.63%
Total runs:
9.2K
Run Growth:
3.2K
Growth Rate:
35.39%
Total runs:
7.7K
Run Growth:
-4.0K
Growth Rate:
-52.89%
Total runs:
5.2K
Run Growth:
-260
Growth Rate:
-4.85%
Total runs:
5.2K
Run Growth:
1.5K
Growth Rate:
28.86%
Total runs:
3.6K
Run Growth:
1.8K
Growth Rate:
51.30%
Total runs:
3.1K
Run Growth:
-4.3K
Growth Rate:
-136.35%
Total runs:
1.9K
Run Growth:
471
Growth Rate:
24.92%
Total runs:
888
Run Growth:
-1.5K
Growth Rate:
-171.16%
Total runs:
842
Run Growth:
-229
Growth Rate:
-27.76%
Total runs:
573
Run Growth:
-1.5K
Growth Rate:
-256.50%
Total runs:
451
Run Growth:
-307
Growth Rate:
-62.53%
Total runs:
431
Run Growth:
394
Growth Rate:
91.42%
Total runs:
341
Run Growth:
-1.8K
Growth Rate:
-558.07%
Total runs:
276
Run Growth:
-192
Growth Rate:
-72.73%
Total runs:
238
Run Growth:
69
Growth Rate:
28.99%
Total runs:
190
Run Growth:
27
Growth Rate:
14.29%
Total runs:
162
Run Growth:
93
Growth Rate:
59.62%
Total runs:
155
Run Growth:
-3
Growth Rate:
-1.94%
Total runs:
149
Run Growth:
-201
Growth Rate:
-133.11%
Total runs:
134
Run Growth:
113
Growth Rate:
85.61%
Total runs:
133
Run Growth:
104
Growth Rate:
78.20%
Total runs:
121
Run Growth:
48
Growth Rate:
39.67%
Total runs:
113
Run Growth:
-38
Growth Rate:
-33.63%
Total runs:
58
Run Growth:
-112
Growth Rate:
-196.49%
Total runs:
56
Run Growth:
21
Growth Rate:
38.18%
Total runs:
34
Run Growth:
17
Growth Rate:
50.00%
Total runs:
32
Run Growth:
15
Growth Rate:
48.39%
Total runs:
21
Run Growth:
-138
Growth Rate:
-657.14%
Total runs:
21
Run Growth:
-91
Growth Rate:
-433.33%
Total runs:
20
Run Growth:
-32
Growth Rate:
-106.67%
Total runs:
19
Run Growth:
5
Growth Rate:
26.32%
Total runs:
19
Run Growth:
-38
Growth Rate:
-126.67%
Total runs:
17
Run Growth:
-16
Growth Rate:
-114.29%
Total runs:
5
Run Growth:
-189
Growth Rate:
-4725.00%