Total runs:
1.9M
Run Growth:
345.0K
Growth Rate:
18.08%
deepset / tinybert-6l-768d-squad2
Model's Last Updated: September 26 2024
question-answering
Introduction of tinybert-6l-768d-squad2
Model Details of tinybert-6l-768d-squad2
Overview
Language model:
deepset/tinybert-6L-768D-squad2
Language:
English
Training data:
SQuAD 2.0 training set x 20 augmented + SQuAD 2.0 training set without augmentation
Eval data:
SQuAD 2.0 dev set
Infrastructure
: 1x V100 GPU
Published
: Dec 8th, 2021
Details
- haystack's intermediate layer and prediction layer distillation features were used for training (based on TinyBERT ). deepset/bert-base-uncased-squad2 was used as the teacher model and huawei-noah/TinyBERT_General_6L_768D was used as the student model.
Hyperparameters
Intermediate layer distillation
batch_size = 26
n_epochs = 5
max_seq_len = 384
learning_rate = 5e-5
lr_schedule = LinearWarmup
embeds_dropout_prob = 0.1
temperature = 1
Prediction layer distillation
batch_size = 26
n_epochs = 5
max_seq_len = 384
learning_rate = 3e-5
lr_schedule = LinearWarmup
embeds_dropout_prob = 0.1
temperature = 1
distillation_loss_weight = 1.0
Performance
"exact": 71.87736882001179
"f1": 76.36111895973675
Authors
-
Timo Möller:
timo.moeller [at] deepset.ai -
Julian Risch:
julian.risch [at] deepset.ai -
Malte Pietsch:
malte.pietsch [at] deepset.ai -
Michel Bartels:
michel.bartels [at] deepset.ai
About us
Our focus: Industry specific language models & large scale QA systems.
Some of our work:
- German BERT (aka "bert-base-german-cased")
- GermanQuAD and GermanDPR datasets and models (aka "gelectra-base-germanquad", "gbert-base-germandpr")
- FARM
- Haystack
Get in touch: Twitter | LinkedIn | Discord | GitHub Discussions | Website
By the way: we're hiring!
Runs of deepset tinybert-6l-768d-squad2 on huggingface.co
292
Total runs
0
24-hour runs
22
3-day runs
155
7-day runs
-31
30-day runs
More Information About tinybert-6l-768d-squad2 huggingface.co Model
Url of tinybert-6l-768d-squad2
tinybert-6l-768d-squad2 huggingface.co Url
Provider of tinybert-6l-768d-squad2 huggingface.co
Other API from deepset
Updated: September 24 2024
Total runs:
224.2K
Run Growth:
36.0K
Growth Rate:
16.07%
Total runs:
104.0K
Run Growth:
26.2K
Growth Rate:
25.21%
Total runs:
51.3K
Run Growth:
42.2K
Growth Rate:
82.10%
Total runs:
42.1K
Run Growth:
32.7K
Growth Rate:
77.54%
Total runs:
38.0K
Run Growth:
13.0K
Growth Rate:
34.12%
Total runs:
31.8K
Run Growth:
-66.4K
Growth Rate:
-208.71%
Total runs:
27.6K
Run Growth:
3.7K
Growth Rate:
13.44%
Total runs:
13.2K
Run Growth:
-1.0M
Growth Rate:
-7909.41%
Total runs:
12.3K
Run Growth:
7.3K
Growth Rate:
59.11%
Total runs:
11.1K
Run Growth:
2.8K
Growth Rate:
25.31%
Total runs:
10.5K
Run Growth:
8.2K
Growth Rate:
78.18%
Total runs:
5.5K
Run Growth:
930
Growth Rate:
16.84%
Total runs:
3.9K
Run Growth:
2.4K
Growth Rate:
60.02%
Total runs:
3.4K
Run Growth:
-17
Growth Rate:
-0.49%
Total runs:
1.9K
Run Growth:
575
Growth Rate:
29.81%
Total runs:
1.3K
Run Growth:
674
Growth Rate:
50.79%
Total runs:
1.2K
Run Growth:
83
Growth Rate:
6.97%
Total runs:
1.0K
Run Growth:
283
Growth Rate:
27.99%
Total runs:
707
Run Growth:
-1.3K
Growth Rate:
-184.16%
Total runs:
679
Run Growth:
297
Growth Rate:
43.74%
Total runs:
545
Run Growth:
521
Growth Rate:
95.60%
Total runs:
484
Run Growth:
-204
Growth Rate:
-42.15%
Total runs:
463
Run Growth:
329
Growth Rate:
71.06%
Total runs:
317
Run Growth:
309
Growth Rate:
97.48%
Total runs:
296
Run Growth:
266
Growth Rate:
89.86%
Total runs:
271
Run Growth:
200
Growth Rate:
73.80%
Total runs:
222
Run Growth:
77
Growth Rate:
34.68%
Total runs:
221
Run Growth:
207
Growth Rate:
93.67%
Total runs:
220
Run Growth:
214
Growth Rate:
97.27%
Total runs:
215
Run Growth:
10
Growth Rate:
4.65%
Total runs:
163
Run Growth:
141
Growth Rate:
86.50%
Total runs:
162
Run Growth:
-32
Growth Rate:
-19.75%
Total runs:
144
Run Growth:
117
Growth Rate:
81.25%
Total runs:
137
Run Growth:
119
Growth Rate:
86.86%
Total runs:
128
Run Growth:
96
Growth Rate:
75.00%
Total runs:
111
Run Growth:
103
Growth Rate:
92.79%
Total runs:
105
Run Growth:
37
Growth Rate:
35.24%
Total runs:
104
Run Growth:
61
Growth Rate:
58.65%
Total runs:
67
Run Growth:
42
Growth Rate:
62.69%
Total runs:
60
Run Growth:
41
Growth Rate:
68.33%
Total runs:
58
Run Growth:
-60
Growth Rate:
-103.45%
Total runs:
24
Run Growth:
6
Growth Rate:
25.00%
Total runs:
19
Run Growth:
-171
Growth Rate:
-900.00%