Introduction of MultiIndicQuestionGenerationUnified
Model Details of MultiIndicQuestionGenerationUnified
MultiIndicQuestionGenerationUnified
MultiIndicQuestionGenerationUnified is a multilingual, sequence-to-sequence pre-trained model, a
IndicBART
checkpoint fine-tuned on the 11 languages of
IndicQuestionGeneration
dataset. For fine-tuning details,
see the
paper
. You can use MultiIndicQuestionGenerationUnified to build question generation applications for Indian languages by fine-tuning the model with supervised training data for the question generation task. Some salient features of the MultiIndicQuestionGenerationUnified are:
Supported languages: Assamese, Bengali, Gujarati, Hindi, Marathi, Oriya, Punjabi, Kannada, Malayalam, Tamil, and Telugu. Not all of these languages are supported by mBART50 and mT5.
The model is much smaller than the mBART and mT5(-base) models, so less computationally expensive for fine-tuning and decoding.
Fine-tuned on large Indic language corpora (770 K examples).
All languages have been represented in Devanagari script to encourage transfer learning among the related languages.
You can read more about MultiIndicQuestionGenerationUnified in this
paper
.
Using this model in
transformers
from transformers import MBartForConditionalGeneration, AutoModelForSeq2SeqLM
from transformers import AlbertTokenizer, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("ai4bharat/MultiIndicQuestionGenerationUnified", do_lower_case=False, use_fast=False, keep_accents=True)
# Or use tokenizer = AlbertTokenizer.from_pretrained("ai4bharat/MultiIndicQuestionGenerationUnified", do_lower_case=False, use_fast=False, keep_accents=True)
model = AutoModelForSeq2SeqLM.from_pretrained("ai4bharat/MultiIndicQuestionGenerationUnified")
# Or use model = MBartForConditionalGeneration.from_pretrained("ai4bharat/MultiIndicQuestionGenerationUnified")
# Some initial mapping
bos_id = tokenizer._convert_token_to_id_with_added_voc("<s>")
eos_id = tokenizer._convert_token_to_id_with_added_voc("</s>")
pad_id = tokenizer._convert_token_to_id_with_added_voc("<pad>")
# To get lang_id use any of ['<2as>', '<2bn>', '<2gu>', '<2hi>', '<2kn>', '<2ml>', '<2mr>', '<2or>', '<2pa>', '<2ta>', '<2te>']
# First tokenize the input and outputs. The format below is how IndicBART was trained so the input should be "Sentence </s> <2xx>" where xx is the language code. Similarly, the output should be "<2yy> Sentence </s>".
inp = tokenizer("7 फरवरी, 2016 [SEP] खेल 7 फरवरी, 2016 को कैलिफोर्निया के सांता क्लारा में सैन फ्रांसिस्को खाड़ी क्षेत्र में लेवी स्टेडियम में खेला गया था।</s><2hi>", add_special_tokens=False, return_tensors="pt", padding=True).input_ids
out = tokenizer("<2hi> सुपर बाउल किस दिन खेला गया? </s>", add_special_tokens=False, return_tensors="pt", padding=True).input_ids
model_outputs=model(input_ids=inp, decoder_input_ids=out[:,0:-1], labels=out[:,1:])
# For loss
model_outputs.loss ## This is not label smoothed.
# For logits
model_outputs.logits
# For generation. Pardon the messiness. Note the decoder_start_token_id.
model.eval() # Set dropouts to zero
model_output=model.generate(inp, use_cache=True,no_repeat_ngram_size=3,encoder_no_repeat_ngram_size=3, num_beams=4, max_length=20, min_length=1, early_stopping=True, pad_token_id=pad_id, bos_token_id=bos_id, eos_token_id=eos_id, decoder_start_token_id=tokenizer._convert_token_to_id_with_added_voc("<2hi>"))
# Decode to get output strings
decoded_output=tokenizer.decode(model_output[0], skip_special_tokens=True, clean_up_tokenization_spaces=False)
print(decoded_output) # कब खेला जाएगा पहला मैच?
# Disclaimer
Note that if your output language is not Hindi or Marathi, you should convert its script from Devanagari to the desired language using the [Indic NLP Library](https://github.com/AI4Bharat/indic-bart/blob/main/indic_scriptmap.py).
Note:
If you wish to use any language written in a non-Devanagari script, then you should first convert it to Devanagari using the
Indic NLP Library
. After you get the output, you should convert it back into the original script.
Benchmarks
Scores on the
IndicQuestionGeneration
test sets are as follows:
Language
RougeL
as
20.48
bn
26.63
gu
27.71
hi
35.38
kn
23.56
ml
22.17
mr
23.52
or
25.25
pa
32.10
ta
22.98
te
25.67
Citation
If you use this model, please cite the following paper:
@inproceedings{Kumar2022IndicNLGSM,
title={IndicNLG Suite: Multilingual Datasets for Diverse NLG Tasks in Indic Languages},
author={Aman Kumar and Himani Shrotriya and Prachi Sahu and Raj Dabre and Ratish Puduppully and Anoop Kunchukuttan and Amogh Mishra and Mitesh M. Khapra and Pratyush Kumar},
year={2022},
url = "https://arxiv.org/abs/2203.05437"
}
License
The model is available under the MIT License.
Runs of ai4bharat MultiIndicQuestionGenerationUnified on huggingface.co
35
Total runs
0
24-hour runs
2
3-day runs
3
7-day runs
27
30-day runs
More Information About MultiIndicQuestionGenerationUnified huggingface.co Model
MultiIndicQuestionGenerationUnified huggingface.co is an AI model on huggingface.co that provides MultiIndicQuestionGenerationUnified's model effect (), which can be used instantly with this ai4bharat MultiIndicQuestionGenerationUnified model. huggingface.co supports a free trial of the MultiIndicQuestionGenerationUnified model, and also provides paid use of the MultiIndicQuestionGenerationUnified. Support call MultiIndicQuestionGenerationUnified model through api, including Node.js, Python, http.
MultiIndicQuestionGenerationUnified huggingface.co is an online trial and call api platform, which integrates MultiIndicQuestionGenerationUnified's modeling effects, including api services, and provides a free online trial of MultiIndicQuestionGenerationUnified, you can try MultiIndicQuestionGenerationUnified online for free by clicking the link below.
ai4bharat MultiIndicQuestionGenerationUnified online free url in huggingface.co:
MultiIndicQuestionGenerationUnified is an open source model from GitHub that offers a free installation service, and any user can find MultiIndicQuestionGenerationUnified on GitHub to install. At the same time, huggingface.co provides the effect of MultiIndicQuestionGenerationUnified install, users can directly use MultiIndicQuestionGenerationUnified installed effect in huggingface.co for debugging and trial. It also supports api for free installation.
MultiIndicQuestionGenerationUnified install url in huggingface.co: