Indic Parler-TTS
is a multilingual Indic extension of
Parler-TTS Mini
.
It is a fine-tuned version of
Indic Parler-TTS Pretrained
, trained on a
1,806 hours
multilingual Indic and English dataset.
Indic Parler-TTS Mini
can officially speak in 20 Indic languages, making it comprehensive for regional language technologies, and in English. The
21 languages
supported are: Assamese, Bengali, Bodo, Dogri, English, Gujarati, Hindi, Kannada, Konkani, Maithili, Malayalam, Manipuri, Marathi, Nepali, Odia, Sanskrit, Santali, Sindhi, Tamil, Telugu, and Urdu.
Thanks to its
better prompt tokenizer
, it can easily be extended to other languages. This tokenizer has a larger vocabulary and handles byte fallback, which simplifies multilingual training.
🚨 This work is the result of a collaboration between the
HuggingFace audio team
and the
AI4Bharat
team
. 🚨
Caption
- A detailed description of how the speech should sound, e.g., "Leela speaks in a high-pitched, fast-paced, and cheerful tone, full of energy and happiness. The recording is very high quality with no background noise."
Unofficial support
: Chhattisgarhi, Kashmiri, Punjabi.
Speaker Diversity
69 unique voices
across the supported languages.
Supported languages have a set of
recommended voices
optimized for naturalness and intelligibility.
Emotion Rendering
10 languages
officially support emotion-specific prompts: Assamese, Bengali, Bodo, Dogri, Kannada, Malayalam, Marathi, Sanskrit, Nepali, and Tamil.
Emotion support for other languages exists but has not been extensively tested.
Available emotions
include: Command, Anger, Narration, Conversation, Disgust, Fear, Happy, Neutral, Proper Noun, News, Sad, and Surprise.
Accent Flexibility
The model
officially supports Indian English accents
through its English voices, providing clear and natural speech.
For other accents, the model allows customization by specifying accent details, such as "A male British speaker" or "A female American speaker," using style transfer for more dynamic and personalized outputs.
Customizable Output
Indic Parler-TTS offers precise control over various speech characteristics using the
caption
input:
Background Noise
: Adjust the noise level in the audio, from clear to slightly noisy environments.
Reverberation
: Control the perceived distance of the voice, from close-sounding to distant-sounding speech.
Expressivity
: Specify how dynamic or monotone the speech should be, ranging from expressive to slightly expressive or monotone.
Pitch
: Modify the pitch of the speech, including high, low, or balanced tones.
Speaking Rate
: Change the speaking rate, from slow to fast.
Voice Quality
: Control the overall clarity and naturalness of the speech, adjusting from basic to refined voice quality.
🎲 Random voice
🚨 Unlike previous versions of Parler-TTS, here we use two tokenizers - one for the prompt and one for the description. 🚨
Indic Parler-TTS
has been trained to generate speech with features that can be controlled with a simple text prompt, for example:
import torch
from parler_tts import ParlerTTSForConditionalGeneration
from transformers import AutoTokenizer
import soundfile as sf
device = "cuda:0"if torch.cuda.is_available() else"cpu"
model = ParlerTTSForConditionalGeneration.from_pretrained("ai4bharat/indic-parler-tts").to(device)
tokenizer = AutoTokenizer.from_pretrained("ai4bharat/indic-parler-tts")
description_tokenizer = AutoTokenizer.from_pretrained(model.config.text_encoder._name_or_path)
prompt = "Hey, how are you doing today?"
description = "A female speaker with a British accent delivers a slightly expressive and animated speech with a moderate speed and pitch. The recording is of very high quality, with the speaker's voice sounding clear and very close up."
input_ids = description_tokenizer(description, return_tensors="pt").input_ids.to(device)
prompt_input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)
generation = model.generate(input_ids=input_ids, prompt_input_ids=prompt_input_ids)
audio_arr = generation.cpu().numpy().squeeze()
sf.write("indic_tts_out.wav", audio_arr, model.config.sampling_rate)
Indic Parler-TTS provides highly effective control over key aspects of speech synthesis using descriptive captions. Below is a summary of what each control parameter can achieve:
Control Type
Capabilities
Background Noise
Adjusts the level of background noise, supporting clear and slightly noisy environments.
Reverberation
Controls the perceived distance of the speaker’s voice, allowing close or distant sounds.
Expressivity
Modulates the emotional intensity of speech, from monotone to highly expressive.
Pitch
Varies the pitch to achieve high, low, or moderate tonal output.
Speaking Rate
Changes the speed of speech delivery, ranging from slow to fast-paced.
Speech Quality
Improves or degrades the overall audio clarity, supporting basic to refined outputs.
🌍 Switching languages
The model automatically adapts to the language it detects in the prompt. You don't need to specify the language you want to use. For example, to switch to Hindi, simply use an Hindi prompt:
import torch
from parler_tts import ParlerTTSForConditionalGeneration
from transformers import AutoTokenizer
import soundfile as sf
device = "cuda:0"if torch.cuda.is_available() else"cpu"
model = ParlerTTSForConditionalGeneration.from_pretrained("ai4bharat/indic-parler-tts").to(device)
tokenizer = AutoTokenizer.from_pretrained("ai4bharat/indic-parler-tts")
description_tokenizer = AutoTokenizer.from_pretrained(model.config.text_encoder._name_or_path)
prompt = "अरे, तुम आज कैसे हो?"
description = "A female speaker delivers a slightly expressive and animated speech with a moderate speed and pitch. The recording is of very high quality, with the speaker's voice sounding clear and very close up."
input_ids = description_tokenizer(description, return_tensors="pt").input_ids.to(device)
prompt_input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)
generation = model.generate(input_ids=input_ids, prompt_input_ids=prompt_input_ids)
audio_arr = generation.cpu().numpy().squeeze()
sf.write("indic_tts_out.wav", audio_arr, model.config.sampling_rate)
🎯 Using a specific speaker
To ensure speaker consistency across generations, this checkpoint was also trained on pre-determined speakers, characterized by name (e.g. Rohit, Karan, Leela, Maya, Sita, ...).
To take advantage of this, simply adapt your text description to specify which speaker to use:
Divya's voice is monotone yet slightly fast in delivery, with a very close recording that almost has no background noise.
import torch
from parler_tts import ParlerTTSForConditionalGeneration
from transformers import AutoTokenizer
import soundfile as sf
device = "cuda:0"if torch.cuda.is_available() else"cpu"
model = ParlerTTSForConditionalGeneration.from_pretrained("ai4bharat/indic-parler-tts").to(device)
tokenizer = AutoTokenizer.from_pretrained("ai4bharat/indic-parler-tts")
description_tokenizer = AutoTokenizer.from_pretrained(model.config.text_encoder._name_or_path)
prompt = "अरे, तुम आज कैसे हो?"
description = "Divya's voice is monotone yet slightly fast in delivery, with a very close recording that almost has no background noise."
input_ids = description_tokenizer(description, return_tensors="pt").input_ids.to(device)
prompt_input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)
generation = model.generate(input_ids=input_ids, prompt_input_ids=prompt_input_ids)
audio_arr = generation.cpu().numpy().squeeze()
sf.write("indic_tts_out.wav", audio_arr, model.config.sampling_rate)
The model includes
69 speakers
across 18 officially supported languages, with each language having a set of recommended voices for optimal performance. Below is a table summarizing the available speakers for each language, along with the recommended ones.
We've set up an
inference guide
to make generation faster. Think SDPA, torch.compile, batching and streaming!
Include the term "very clear audio" to generate the highest quality audio, and "very noisy audio" for high levels of background noise
Punctuation can be used to control the prosody of the generations, e.g. use commas to add small breaks in speech
The remaining speech features (gender, speaking rate, pitch and reverberation) can be controlled directly through the prompt
Some Description Examples
Aditi - Slightly High-Pitched, Expressive Tone
:
"Aditi speaks with a slightly higher pitch in a close-sounding environment. Her voice is clear, with subtle emotional depth and a normal pace, all captured in high-quality recording."
Sita - Rapid, Slightly Monotone
:
"Sita speaks at a fast pace with a slightly low-pitched voice, captured clearly in a close-sounding environment with excellent recording quality."
Tapan - Male, Moderate Pace, Slightly Monotone
:
"Tapan speaks at a moderate pace with a slightly monotone tone. The recording is clear, with a close sound and only minimal ambient noise."
Sunita - High-Pitched, Happy Tone
:
"Sunita speaks with a high pitch in a close environment. Her voice is clear, with slight dynamic changes, and the recording is of excellent quality."
Karan - High-Pitched, Positive Tone
:
"Karan’s high-pitched, engaging voice is captured in a clear, close-sounding recording. His slightly slower delivery conveys a positive tone."
Amrita - High-Pitched, Flat Tone
:
"Amrita speaks with a high pitch at a slow pace. Her voice is clear, with excellent recording quality and only moderate background noise."
Aditi - Slow, Slightly Expressive
:
"Aditi speaks slowly with a high pitch and expressive tone. The recording is clear, showcasing her energetic and emotive voice."
Young Male Speaker, American Accent
:
"A young male speaker with a high-pitched American accent delivers speech at a slightly fast pace in a clear, close-sounding recording."
Bikram - High-Pitched, Urgent Tone
:
"Bikram speaks with a higher pitch and fast pace, conveying urgency. The recording is clear and intimate, with great emotional depth."
Anjali - High-Pitched, Neutral Tone
:
"Anjali speaks with a high pitch at a normal pace in a clear, close-sounding environment. Her neutral tone is captured with excellent audio quality."
📐 Evaluation
Indic Parler-TTS has been evaluated using a MOS-like framework by native and non-native speakers. The results highlight its exceptional performance in generating natural and intelligible speech, especially for native speakers of Indian languages.
NSS
stands for
Native Speaker Score
:
Language
NSS Pretrained (%)
NSS Finetuned (%)
Highlights
Assamese
82.56 ± 1.80
87.36 ± 1.81
Clear, natural synthesis with excellent expressiveness.
Bengali
77.41 ± 2.14
86.16 ± 1.85
High-quality outputs with smooth intonation.
Bodo
90.83 ± 4.54
94.47 ± 4.12
Near-perfect accuracy for a lesser-resourced language.
Dogri
82.61 ± 4.98
88.80 ± 3.57
Robust and consistent synthesis for Dogri.
Gujarati
75.28 ± 1.94
75.36 ± 1.78
Strong clarity and naturalness even for smaller languages.
Hindi
83.43 ± 1.53
84.79 ± 2.09
Reliable and expressive outputs for India's most widely spoken language.
Kannada
77.97 ± 3.43
88.17 ± 2.81
Highly natural and accurate voices for Kannada.
Konkani
87.20 ± 3.58
76.60 ± 4.14
Produces clear and natural outputs for diverse speakers.
Description
:
The model was fine-tuned on a subset of the dataset used to train the pre-trained version:
Indic-Parler Dataset
, a large-scale multilingual speech corpus designed to train the
Indic Parler-TTS
model.
Key Statistics
:
Dataset
Duration (hrs)
Languages Covered
No. of Utterances
License
GLOBE
535.0
1
581,725
CC V1
IndicTTS
382.0
12
220,606
CC BY 4.0
LIMMITS
568.0
7
246,008
CC BY 4.0
Rasa
288.0
9
155,734
CC BY 4.0
Languages Covered
:
The dataset supports
16 official languages
of India, along with English and Chhattisgarhi, making it comprehensive for regional language technologies. These languages include Assamese, Bengali, Bodo, Chhattisgarhi, Dogri, English, Gujarati, Hindi, Kannada, Malayalam, Manipuri, Marathi, Nepali, Odia, Punjabi, Sanskrit, Tamil and Telugu.
Language-Wise Data Breakdown
:
Here’s the table combining the duration (hours) and the number of utterances from the provided stats:
Language
Duration (hrs)
No. of Utterances
Assamese
69.78
41,210
Bengali
140.04
70,305
Bodo
49.14
27,012
Chhattisgarhi
80.11
38,148
Dogri
16.14
7,823
English
802.81
735,482
Gujarati
21.24
5,679
Hindi
107.00
46,135
Kannada
125.01
54,575
Malayalam
25.21
14,988
Manipuri
20.77
19,232
Marathi
122.47
54,894
Nepali
28.65
16,016
Odia
19.18
11,558
Punjabi
11.07
6,892
Sanskrit
19.91
8,720
Tamil
52.25
29,204
Telugu
95.91
37,405
Citation
If you found this repository useful, please consider citing this work and also the original Stability AI paper:
@misc{lacombe-etal-2024-parler-tts,
author = {Yoach Lacombe and Vaibhav Srivastav and Sanchit Gandhi},
title = {Parler-TTS},
year = {2024},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/huggingface/parler-tts}}
}
@misc{lyth2024natural,
title={Natural language guidance of high-fidelity text-to-speech with synthetic annotations},
author={Dan Lyth and Simon King},
year={2024},
eprint={2402.01912},
archivePrefix={arXiv},
primaryClass={cs.SD}
}
License
This model is permissively licensed under the Apache 2.0 license.
Runs of ai4bharat indic-parler-tts on huggingface.co
19.1K
Total runs
299
24-hour runs
2.3K
3-day runs
4.3K
7-day runs
38.4K
30-day runs
More Information About indic-parler-tts huggingface.co Model
indic-parler-tts huggingface.co is an AI model on huggingface.co that provides indic-parler-tts's model effect (), which can be used instantly with this ai4bharat indic-parler-tts model. huggingface.co supports a free trial of the indic-parler-tts model, and also provides paid use of the indic-parler-tts. Support call indic-parler-tts model through api, including Node.js, Python, http.
indic-parler-tts huggingface.co is an online trial and call api platform, which integrates indic-parler-tts's modeling effects, including api services, and provides a free online trial of indic-parler-tts, you can try indic-parler-tts online for free by clicking the link below.
ai4bharat indic-parler-tts online free url in huggingface.co:
indic-parler-tts is an open source model from GitHub that offers a free installation service, and any user can find indic-parler-tts on GitHub to install. At the same time, huggingface.co provides the effect of indic-parler-tts install, users can directly use indic-parler-tts installed effect in huggingface.co for debugging and trial. It also supports api for free installation.