Total runs:
1.4M
Run Growth:
-288.0K
Growth Rate:
-20.67%
dbmdz / bert-base-historic-dutch-cased
Model's Last Updated: November 15 2024
fill-mask
Introduction of bert-base-historic-dutch-cased
Model Details of bert-base-historic-dutch-cased
Language Model for Historic Dutch
In this repository we open source a language model for Historic Dutch, trained on the Delpher Corpus , that include digitized texts from Dutch newspapers, ranging from 1618 to 1879.
Changelog
- 13.12.2021: Initial version of this repository.
Model Zoo
The following models for Historic Dutch are available on the Hugging Face Model Hub:
| Model identifier | Model Hub link |
|---|---|
dbmdz/bert-base-historic-dutch-cased
|
here |
Stats
The download urls for all archives can be found here .
We then used the awesome
alto-tools
from
this
repository to extract plain text. The following table shows the size overview per year range:
| Period | Extracted plain text size |
|---|---|
| 1618-1699 | 170MB |
| 1700-1709 | 103MB |
| 1710-1719 | 65MB |
| 1720-1729 | 137MB |
| 1730-1739 | 144MB |
| 1740-1749 | 188MB |
| 1750-1759 | 171MB |
| 1760-1769 | 235MB |
| 1770-1779 | 271MB |
| 1780-1789 | 414MB |
| 1790-1799 | 614MB |
| 1800-1809 | 734MB |
| 1810-1819 | 807MB |
| 1820-1829 | 987MB |
| 1830-1839 | 1.7GB |
| 1840-1849 | 2.2GB |
| 1850-1854 | 1.3GB |
| 1855-1859 | 1.7GB |
| 1860-1864 | 2.0GB |
| 1865-1869 | 2.3GB |
| 1870-1874 | 1.9GB |
| 1875-1876 | 867MB |
| 1877-1879 | 1.9GB |
The total training corpus consists of 427,181,269 sentences and 3,509,581,683 tokens (counted via
wc
),
resulting in a total corpus size of 21GB.
The following figure shows an overview of the number of chars per year distribution:
Language Model Pretraining
We use the official BERT implementation using the following command to train the model:
python3 run_pretraining.py --input_file gs://delpher-bert/tfrecords/*.tfrecord \
--output_dir gs://delpher-bert/bert-base-historic-dutch-cased \
--bert_config_file ./config.json \
--max_seq_length=512 \
--max_predictions_per_seq=75 \
--do_train=True \
--train_batch_size=128 \
--num_train_steps=3000000 \
--learning_rate=1e-4 \
--save_checkpoints_steps=100000 \
--keep_checkpoint_max=20 \
--use_tpu=True \
--tpu_name=electra-2 \
--num_tpu_cores=32
We train the model for 3M steps using a total batch size of 128 on a v3-32 TPU. The pretraining loss curve can be seen in the next figure:
Evaluation
We evaluate our model on the preprocessed Europeana NER dataset for Dutch, that was presented in the "Data Centric Domain Adaptation for Historical Text with OCR Errors" paper.
The data is available in their repository. We perform a hyper-parameter search for:
-
Batch sizes:
[4, 8] -
Learning rates:
[3e-5, 5e-5] -
Number of epochs:
[5, 10]
and report averaged F1-Score over 5 runs with different seeds. We also include hmBERT as baseline model.
Results:
| Model | F1-Score (Dev / Test) |
|---|---|
| hmBERT | (82.73) / 81.34 |
| Maerz et al. (2021) | - / 84.2 |
| Ours | (89.73) / 87.45 |
License
All models are licensed under MIT .
Acknowledgments
Research supported with Cloud TPUs from Google's TPU Research Cloud (TRC) program, previously known as TensorFlow Research Cloud (TFRC). Many thanks for providing access to the TRC ❤️
We thank Clemens Neudecker for maintaining the amazing ALTO tools that were used for parsing the Delpher Corpus XML files.
Thanks to the generous support from the Hugging Face team, it is possible to download both cased and uncased models from their S3 storage 🤗
Runs of dbmdz bert-base-historic-dutch-cased on huggingface.co
117
Total runs
0
24-hour runs
99
3-day runs
104
7-day runs
92
30-day runs
More Information About bert-base-historic-dutch-cased huggingface.co Model
Url of bert-base-historic-dutch-cased
bert-base-historic-dutch-cased huggingface.co Url
Provider of bert-base-historic-dutch-cased huggingface.co
Other API from dbmdz
Updated: September 07 2023
Total runs:
86.2K
Run Growth:
-201
Growth Rate:
-0.23%
Total runs:
68.9K
Run Growth:
-16.8K
Growth Rate:
-23.27%
Total runs:
65.9K
Run Growth:
25.4K
Growth Rate:
39.83%
Total runs:
48.1K
Run Growth:
11.5K
Growth Rate:
23.63%
Total runs:
42.5K
Run Growth:
35.0K
Growth Rate:
85.08%
Total runs:
38.3K
Run Growth:
23.0K
Growth Rate:
61.06%
Total runs:
26.4K
Run Growth:
9.3K
Growth Rate:
36.52%
Total runs:
25.3K
Run Growth:
1.9K
Growth Rate:
7.66%
Total runs:
9.7K
Run Growth:
2.2K
Growth Rate:
23.87%
Total runs:
6.8K
Run Growth:
521
Growth Rate:
7.75%
Total runs:
4.7K
Run Growth:
2.0K
Growth Rate:
43.25%
Total runs:
3.7K
Run Growth:
-1.5K
Growth Rate:
-41.94%
Total runs:
1.5K
Run Growth:
26
Growth Rate:
1.67%
Total runs:
1.4K
Run Growth:
-2.4K
Growth Rate:
-190.95%
Total runs:
1.3K
Run Growth:
774
Growth Rate:
58.77%
Total runs:
980
Run Growth:
-218
Growth Rate:
-21.80%
Total runs:
567
Run Growth:
-219
Growth Rate:
-39.82%
Total runs:
526
Run Growth:
-195
Growth Rate:
-40.46%
Total runs:
476
Run Growth:
263
Growth Rate:
57.80%
Total runs:
286
Run Growth:
23
Growth Rate:
8.04%
Total runs:
262
Run Growth:
221
Growth Rate:
77.00%
Total runs:
262
Run Growth:
189
Growth Rate:
71.59%
Total runs:
237
Run Growth:
-182
Growth Rate:
-78.79%
Total runs:
183
Run Growth:
124
Growth Rate:
67.39%
Total runs:
164
Run Growth:
122
Growth Rate:
74.39%
Total runs:
149
Run Growth:
-190
Growth Rate:
-128.38%
Total runs:
138
Run Growth:
125
Growth Rate:
90.58%
Total runs:
124
Run Growth:
793
Growth Rate:
98.63%
Total runs:
121
Run Growth:
6
Growth Rate:
30.00%
Total runs:
121
Run Growth:
58
Growth Rate:
48.74%
Total runs:
113
Run Growth:
-49
Growth Rate:
-41.53%
Total runs:
110
Run Growth:
63
Growth Rate:
57.80%
Total runs:
105
Run Growth:
93
Growth Rate:
88.57%
Total runs:
88
Run Growth:
55
Growth Rate:
62.50%
Total runs:
77
Run Growth:
6
Growth Rate:
8.33%
Total runs:
46
Run Growth:
20
Growth Rate:
43.48%
Total runs:
29
Run Growth:
21
Growth Rate:
72.41%
Total runs:
25
Run Growth:
10
Growth Rate:
37.04%
Total runs:
25
Run Growth:
-5
Growth Rate:
-18.52%
Total runs:
22
Run Growth:
-1
Growth Rate:
-4.76%
Total runs:
21
Run Growth:
1
Growth Rate:
4.35%
Total runs:
15
Run Growth:
7
Growth Rate:
46.67%
Total runs:
14
Run Growth:
1
Growth Rate:
7.14%
Total runs:
12
Run Growth:
2
Growth Rate:
16.67%
Total runs:
12
Run Growth:
-2
Growth Rate:
-16.67%
Total runs:
6
Run Growth:
-5
Growth Rate:
-500.00%
Total runs:
5
Run Growth:
-1
Growth Rate:
-20.00%
Total runs:
5
Run Growth:
0
Growth Rate:
0.00%
Total runs:
5
Run Growth:
-12
Growth Rate:
-240.00%
Total runs:
4
Run Growth:
-2
Growth Rate:
-50.00%
Total runs:
2
Run Growth:
-8
Growth Rate:
-400.00%
Total runs:
2
Run Growth:
-53
Growth Rate:
-1766.67%
Total runs:
2
Run Growth:
-4
Growth Rate:
-400.00%
Total runs:
2
Run Growth:
-6
Growth Rate:
-300.00%
Total runs:
2
Run Growth:
-7
Growth Rate:
-350.00%
Total runs:
2
Run Growth:
-3
Growth Rate:
-150.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%