nvidia / canary-1b

Canary 1B

| |

NVIDIA NeMo Canary is a family of multi-lingual multi-tasking models that achieves state-of-the art performance on multiple benchmarks. With 1 billion parameters, Canary-1B supports automatic speech-to-text recognition (ASR) in 4 languages (English, German, French, Spanish) and translation from English to German/French/Spanish and from German/French/Spanish to English with or without punctuation and capitalization (PnC).

Model Architecture

Canary is an encoder-decoder model with FastConformer [1] encoder and Transformer Decoder [2]. With audio features extracted from the encoder, task tokens such as <source language> , <target language> , <task> and <toggle PnC> are fed into the Transformer Decoder to trigger the text generation process. Canary uses a concatenated tokenizer [5] from individual SentencePiece [3] tokenizers of each language, which makes it easy to scale up to more languages. The Canay-1B model has 24 encoder layers and 24 layers of decoder layers in total.

NVIDIA NeMo

To train, fine-tune or Transcribe with Canary, you will need to install NVIDIA NeMo . We recommend you install it after you've installed Cython and latest PyTorch version.

pip install git+https://github.com/NVIDIA/[email protected]#egg=nemo_toolkit[asr]

How to Use this Model

The model is available for use in the NeMo toolkit [4], and can be used as a pre-trained checkpoint for inference or for fine-tuning on another dataset.

Loading the Model

from nemo.collections.asr.models import EncDecMultiTaskModel

# load model
canary_model = EncDecMultiTaskModel.from_pretrained('nvidia/canary-1b')

# update dcode params
decode_cfg = canary_model.cfg.decoding
decode_cfg.beam.beam_size = 1
canary_model.change_decoding_strategy(decode_cfg)

Input Format

Input to Canary can be either a list of paths to audio files or a jsonl manifest file.

If the input is a list of paths, Canary assumes that the audio is English and Transcribes it. I.e., Canary default behaviour is English ASR.

predicted_text = canary_model.transcribe(
    paths2audio_files=['path1.wav', 'path2.wav'],
    batch_size=16,  # batch size to run the inference with
)

To use Canary for transcribing other supported languages or perform Speech-to-Text translation, specify the input as jsonl manifest file, where each line in the file is a dictionary containing the following fields:

# Example of a line in input_manifest.json
{
    "audio_filepath": "/path/to/audio.wav",  # path to the audio file
    "duration": 1000,  # duration of the audio, can be set to `None` if using NeMo main branch
    "taskname": "asr",  # use "s2t_translation" for speech-to-text translation with r1.23, or "ast" if using the NeMo main branch
    "source_lang": "en",  # language of the audio input, set `source_lang`==`target_lang` for ASR, choices=['en','de','es','fr']
    "target_lang": "en",  # language of the text output, choices=['en','de','es','fr']
    "pnc": "yes",  # whether to have PnC output, choices=['yes', 'no']
    "answer": "na", 
}

and then use:

predicted_text = canary_model.transcribe(
    "<path to input manifest file>",
    batch_size=16,  # batch size to run the inference with
)

Automatic Speech-to-text Recognition (ASR)

An example manifest for transcribing English audios can be:

# Example of a line in input_manifest.json
{
    "audio_filepath": "/path/to/audio.wav",  # path to the audio file
    "duration": 1000,  # duration of the audio, can be set to `None` if using NeMo main branch
    "taskname": "asr",  
    "source_lang": "en", # language of the audio input, set `source_lang`==`target_lang` for ASR, choices=['en','de','es','fr']
    "target_lang": "en", # language of the text output, choices=['en','de','es','fr']
    "pnc": "yes",  # whether to have PnC output, choices=['yes', 'no']
    "answer": "na", 
}

Automatic Speech-to-text Translation (AST)

An example manifest for transcribing English audios into German text can be:

# Example of a line in input_manifest.json
{
    "audio_filepath": "/path/to/audio.wav",  # path to the audio file
    "duration": 1000,  # duration of the audio, can be set to `None` if using NeMo main branch
    "taskname": "s2t_translation", # r1.23 only recognizes "s2t_translation", but "ast" is supported if using the NeMo main branch
    "source_lang": "en", # language of the audio input, choices=['en','de','es','fr']
    "target_lang": "de", # language of the text output, choices=['en','de','es','fr']
    "pnc": "yes",  # whether to have PnC output, choices=['yes', 'no']
    "answer": "na" 
}

Alternatively, one can use transcribe_speech.py script to do the same.

python [NEMO_GIT_FOLDER]/examples/asr/transcribe_speech.py 
 pretrained_name="nvidia/canary-1b" 
 audio_dir="<path to audio_directory>" # transcribes all the wav files in audio_directory

python [NEMO_GIT_FOLDER]/examples/asr/transcribe_speech.py 
 pretrained_name="nvidia/canary-1b" 
 dataset_manifest="<path to manifest file>" 

Input

This model accepts single channel (mono) audio sampled at 16000 Hz, along with the task/languages/PnC tags as input.

Output

The model outputs the transcribed/translated text corresponding to the input audio, in the specified target language and with or without punctuation and capitalization.

Training

Canary-1B is trained using the NVIDIA NeMo toolkit [4] for 150k steps with dynamic bucketing and a batch duration of 360s per GPU on 128 NVIDIA A100 80GB GPUs. The model can be trained using this example script and base config .

The tokenizers for these models were built using the text transcripts of the train set with this script .

Datasets

The Canary-1B model is trained on a total of 85k hrs of speech data. It consists of 31k hrs of public data, 20k hrs collected by Suno , and 34k hrs of in-house data.

The constituents of public data are as follows.

English (25.5k hours)

• Librispeech 960 hours
• Fisher Corpus
• Switchboard-1 Dataset
• WSJ-0 and WSJ-1
• National Speech Corpus (Part 1, Part 6)
• VCTK
• VoxPopuli (EN)
• Europarl-ASR (EN)
• Multilingual Librispeech (MLS EN) - 2,000 hour subset
• Mozilla Common Voice (v7.0)
• People's Speech - 12,000 hour subset
• Mozilla Common Voice (v11.0) - 1,474 hour subset

German (2.5k hours)

• Mozilla Common Voice (v12.0) - 800 hour subset
• Multilingual Librispeech (MLS DE) - 1,500 hour subset
• VoxPopuli (DE) - 200 hr subset

Spanish (1.4k hours)

• Mozilla Common Voice (v12.0) - 395 hour subset
• Multilingual Librispeech (MLS ES) - 780 hour subset
• VoxPopuli (ES) - 108 hour subset
• Fisher - 141 hour subset

French (1.8k hours)

• Mozilla Common Voice (v12.0) - 708 hour subset
• Multilingual Librispeech (MLS FR) - 926 hour subset
• VoxPopuli (FR) - 165 hour subset

Performance

In both ASR and AST experiments, predictions were generated using beam search with width 5 and length penalty 1.0.

ASR Performance (w/o PnC)

The ASR performance is measured with word error rate (WER), and we process the groundtruth and predicted text with whisper-normalizer .

WER on MCV-16.1 test set:

WER on MLS test set:

More details on evaluation can be found at HuggingFace ASR Leaderboard

AST Performance

We evaluate AST performance with BLEU score , and use native annotations with punctuation and capitalization in the datasets.

BLEU score on FLEURS test set:

BLEU score on COVOST-v2 test set:

BLEU score on mExpresso test set:

Model Fairness Evaluation

As outlined in the paper "Towards Measuring Fairness in AI: the Casual Conversations Dataset", we assessed the canary-1.1b model for fairness. The model was evaluated on the CausalConversations-v1 dataset, and the results are reported as follows:

Gender Bias:

Age Bias:

(Error rates for fairness evaluation are determined by normalizing both the reference and predicted text, similar to the methods used in the evaluations found at https://github.com/huggingface/open_asr_leaderboard .)

NVIDIA Riva: Deployment

NVIDIA Riva , is an accelerated speech AI SDK deployable on-prem, in all clouds, multi-cloud, hybrid, on edge, and embedded. Additionally, Riva provides:

• World-class out-of-the-box accuracy for the most common languages with model checkpoints trained on proprietary data with hundreds of thousands of GPU-compute hours
• Best in class accuracy with run-time word boosting (e.g., brand and product names) and customization of acoustic model, language model, and inverse text normalization
• Streaming speech recognition, Kubernetes compatible scaling, and enterprise-grade support

Although this model isn’t supported yet by Riva, the list of supported models is here.
Check out Riva live demo .

References

[1] Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition

[2] Attention is all you need

[3] Google Sentencepiece Tokenizer

[4] NVIDIA NeMo Toolkit

[5] Unified Model for Code-Switching Speech Recognition and Language Identification Based on Concatenated Tokenizer

Licence

License to use this model is covered by the CC-BY-NC-4.0 . By downloading the public and release version of the model, you accept the terms and conditions of the CC-BY-NC-4.0 license.