Please use 'Bert' related functions to load this model!
This repository contains the resources in our paper
"Revisiting Pre-trained Models for Chinese Natural Language Processing"
, which will be published in "
Findings of EMNLP
". You can read our camera-ready paper through
ACL Anthology
or
arXiv pre-print
.
MacBERT
is an improved BERT with novel
M
LM
a
s
c
orrection pre-training task, which mitigates the discrepancy of pre-training and fine-tuning.
Instead of masking with [MASK] token, which never appears in the fine-tuning stage,
we propose to use similar words for the masking purpose
. A similar word is obtained by using
Synonyms toolkit (Wang and Hu, 2017)
, which is based on word2vec (Mikolov et al., 2013) similarity calculations. If an N-gram is selected to mask, we will find similar words individually. In rare cases, when there is no similar word, we will degrade to use random word replacement.
Here is an example of our pre-training task.
Example
Original Sentence
we use a language model to predict the probability of the next word.
MLM
we use a language [M] to [M] ##di ##ct the pro [M] ##bility of the next word .
Whole word masking
we use a language [M] to [M] [M] [M] the [M] [M] [M] of the next word .
N-gram masking
we use a [M] [M] to [M] [M] [M] the [M] [M] [M] [M] [M] next word .
MLM as correction
we use a text system to ca ##lc ##ulate the po ##si ##bility of the next word .
Except for the new pre-training task, we also incorporate the following techniques.
Whole Word Masking (WWM)
N-gram masking
Sentence-Order Prediction (SOP)
Note that our MacBERT can be directly replaced with the original BERT as there is no differences in the main neural architecture.
@inproceedings{cui-etal-2020-revisiting,
title = "Revisiting Pre-Trained Models for {C}hinese Natural Language Processing",
author = "Cui, Yiming and
Che, Wanxiang and
Liu, Ting and
Qin, Bing and
Wang, Shijin and
Hu, Guoping",
booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings",
month = nov,
year = "2020",
address = "Online",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/2020.findings-emnlp.58",
pages = "657--668",
}
Runs of hfl chinese-macbert-base on huggingface.co
5.7K
Total runs
-442
24-hour runs
-822
3-day runs
-1.4K
7-day runs
-4.9K
30-day runs
More Information About chinese-macbert-base huggingface.co Model
chinese-macbert-base huggingface.co is an AI model on huggingface.co that provides chinese-macbert-base's model effect (), which can be used instantly with this hfl chinese-macbert-base model. huggingface.co supports a free trial of the chinese-macbert-base model, and also provides paid use of the chinese-macbert-base. Support call chinese-macbert-base model through api, including Node.js, Python, http.
chinese-macbert-base huggingface.co is an online trial and call api platform, which integrates chinese-macbert-base's modeling effects, including api services, and provides a free online trial of chinese-macbert-base, you can try chinese-macbert-base online for free by clicking the link below.
hfl chinese-macbert-base online free url in huggingface.co:
chinese-macbert-base is an open source model from GitHub that offers a free installation service, and any user can find chinese-macbert-base on GitHub to install. At the same time, huggingface.co provides the effect of chinese-macbert-base install, users can directly use chinese-macbert-base installed effect in huggingface.co for debugging and trial. It also supports api for free installation.
chinese-macbert-base install url in huggingface.co: