Total runs:
1.9M
Run Growth:
103.0K
Growth Rate:
5.40%
deepset / gbert-base-germandpr-ctx_encoder
Model's Last Updated: September 26 2024
Introduction of gbert-base-germandpr-ctx_encoder
Model Details of gbert-base-germandpr-ctx_encoder
Overview
Language model:
gbert-base-germandpr
Language:
German
Training data:
GermanDPR train set (~ 56MB)
Eval data:
GermanDPR test set (~ 6MB)
Infrastructure
: 4x V100 GPU
Published
: Apr 26th, 2021
Details
- We trained a dense passage retrieval model with two gbert-base models as encoders of questions and passages.
- The dataset is GermanDPR, a new, German language dataset, which we hand-annotated and published online .
- It comprises 9275 question/answer pairs in the training set and 1025 pairs in the test set. For each pair, there are one positive context and three hard negative contexts.
- As the basis of the training data, we used our hand-annotated GermanQuAD dataset as positive samples and generated hard negative samples from the latest German Wikipedia dump (6GB of raw txt files).
- The data dump was cleaned with tailored scripts, leading to 2.8 million indexed passages from German Wikipedia.
See https://deepset.ai/germanquad for more details and dataset download.
Hyperparameters
batch_size = 40
n_epochs = 20
num_training_steps = 4640
num_warmup_steps = 460
max_seq_len = 32 tokens for question encoder and 300 tokens for passage encoder
learning_rate = 1e-6
lr_schedule = LinearWarmup
embeds_dropout_prob = 0.1
num_hard_negatives = 2
Performance
During training, we monitored the in-batch average rank and the loss and evaluated different batch sizes, numbers of epochs, and number of hard negatives on a dev set split from the train set. The dev split contained 1030 question/answer pairs. Even without thorough hyperparameter tuning, we observed quite stable learning. Multiple restarts with different seeds produced quite similar results. Note that the in-batch average rank is influenced by settings for batch size and number of hard negatives. A smaller number of hard negatives makes the task easier. After fixing the hyperparameters we trained the model on the full GermanDPR train set.
We further evaluated the retrieval performance of the trained model on the full German Wikipedia with the GermanDPR test set as labels. To this end, we converted the GermanDPR test set to SQuAD format. The DPR model drastically outperforms the BM25 baseline with regard to recall@k.
Usage
In haystack
You can load the model in haystack as a retriever for doing QA at scale:
retriever = DensePassageRetriever(
document_store=document_store,
query_embedding_model="deepset/gbert-base-germandpr-question_encoder"
passage_embedding_model="deepset/gbert-base-germandpr-ctx_encoder"
)
Authors
-
Timo Möller:
timo.moeller [at] deepset.ai -
Julian Risch:
julian.risch [at] deepset.ai -
Malte Pietsch:
malte.pietsch [at] deepset.ai
About us
Our focus: Industry specific language models & large scale QA systems.
Some of our work:
- German BERT (aka "bert-base-german-cased")
- GermanQuAD and GermanDPR datasets and models (aka "gelectra-base-germanquad", "gbert-base-germandpr")
- FARM
- Haystack
Get in touch: Twitter | LinkedIn | Website
By the way: we're hiring!
Runs of deepset gbert-base-germandpr-ctx_encoder on huggingface.co
51
Total runs
0
24-hour runs
14
3-day runs
17
7-day runs
34
30-day runs
More Information About gbert-base-germandpr-ctx_encoder huggingface.co Model
Url of gbert-base-germandpr-ctx_encoder
gbert-base-germandpr-ctx_encoder huggingface.co Url
Provider of gbert-base-germandpr-ctx_encoder huggingface.co
Other API from deepset
Updated: September 24 2024
Total runs:
221.8K
Run Growth:
12.9K
Growth Rate:
5.85%
Total runs:
100.5K
Run Growth:
15.3K
Growth Rate:
15.36%
Total runs:
50.7K
Run Growth:
35.7K
Growth Rate:
71.98%
Total runs:
37.2K
Run Growth:
-1.3M
Growth Rate:
-1931.12%
Total runs:
35.8K
Run Growth:
6.0K
Growth Rate:
17.15%
Total runs:
35.4K
Run Growth:
21.4K
Growth Rate:
62.74%
Total runs:
29.3K
Run Growth:
-73.3K
Growth Rate:
-254.69%
Total runs:
27.6K
Run Growth:
4.2K
Growth Rate:
15.21%
Total runs:
9.7K
Run Growth:
4.2K
Growth Rate:
43.76%
Total runs:
9.1K
Run Growth:
-1.4K
Growth Rate:
-16.10%
Total runs:
5.6K
Run Growth:
2.0K
Growth Rate:
38.14%
Total runs:
5.4K
Run Growth:
-458
Growth Rate:
-8.70%
Total runs:
3.6K
Run Growth:
1.9K
Growth Rate:
52.52%
Total runs:
3.2K
Run Growth:
-2.5K
Growth Rate:
-78.85%
Total runs:
1.9K
Run Growth:
500
Growth Rate:
25.47%
Total runs:
990
Run Growth:
202
Growth Rate:
21.15%
Total runs:
883
Run Growth:
13
Growth Rate:
1.45%
Total runs:
610
Run Growth:
-1.5K
Growth Rate:
-242.22%
Total runs:
431
Run Growth:
394
Growth Rate:
91.84%
Total runs:
418
Run Growth:
-28
Growth Rate:
-6.78%
Total runs:
405
Run Growth:
-1.3K
Growth Rate:
-349.61%
Total runs:
377
Run Growth:
-444
Growth Rate:
-118.09%
Total runs:
350
Run Growth:
-86
Growth Rate:
-26.30%
Total runs:
232
Run Growth:
161
Growth Rate:
69.70%
Total runs:
207
Run Growth:
135
Growth Rate:
68.18%
Total runs:
186
Run Growth:
28
Growth Rate:
15.05%
Total runs:
182
Run Growth:
113
Growth Rate:
63.13%
Total runs:
137
Run Growth:
-207
Growth Rate:
-142.76%
Total runs:
132
Run Growth:
111
Growth Rate:
84.73%
Total runs:
126
Run Growth:
90
Growth Rate:
71.43%
Total runs:
115
Run Growth:
-2
Growth Rate:
-1.79%
Total runs:
96
Run Growth:
-136
Growth Rate:
-158.14%
Total runs:
63
Run Growth:
37
Growth Rate:
58.73%
Total runs:
60
Run Growth:
-86
Growth Rate:
-148.28%
Total runs:
55
Run Growth:
22
Growth Rate:
40.74%
Total runs:
28
Run Growth:
10
Growth Rate:
34.48%
Total runs:
21
Run Growth:
-100
Growth Rate:
-476.19%
Total runs:
21
Run Growth:
-17
Growth Rate:
-80.95%
Total runs:
20
Run Growth:
6
Growth Rate:
31.58%
Total runs:
20
Run Growth:
-40
Growth Rate:
-200.00%
Total runs:
18
Run Growth:
-12
Growth Rate:
-70.59%
Total runs:
13
Run Growth:
-54
Growth Rate:
-284.21%
Total runs:
6
Run Growth:
-188
Growth Rate:
-3760.00%