--- license: apache-2.0 language: - zh - yue - en library_name: onnx-asr pipeline_tag: automatic-speech-recognition tags: - automatic-speech-recognition - onnx - onnx-asr - paraformer - funasr - cantonese base_model: - csukuangfj/sherpa-onnx-paraformer-trilingual-zh-cantonese-en --- # Paraformer trilingual ZH / Cantonese / EN ONNX A SeACo-Paraformer trained on Mandarin, Cantonese (Yue) and English for onnx-asr. This is the strongest available Cantonese model in this collection: it is character identical to native FunASR on every Cantonese and Mandarin test clip, fp32 and int8 alike, and it replaces `OpenVoiceOS/whisper-small` as the Cantonese default, since that model is a general multilingual Whisper checkpoint with no Cantonese-specific training. ## What this is Paraformer is the Alibaba [FunASR](https://github.com/modelscope/FunASR) offline non-autoregressive recognizer. A SAN-M encoder reads the audio, a CIF predictor decides how many tokens the utterance has, and a single pass decoder emits all of them at once. There is no decoding loop, so one forward pass gives the transcript. This checkpoint is a SeACo-Paraformer, the hotword-biasing variant of Paraformer. The graph exposes two extra outputs used by the bias pathway (`decoder_hidden`, `pre_acoustic_embeds`), but the `paraformer` model type in onnx-asr only requests `logits` and `token_num` by name, so it loads and runs unmodified, with no hotword biasing applied. Its `speech` / `speech_lengths` inputs and `logits` / `token_num` outputs match the plain Paraformer contract exactly. The ONNX graphs here are copied byte for byte from the [sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx) export by [csukuangfj](https://huggingface.co/csukuangfj/sherpa-onnx-paraformer-trilingual-zh-cantonese-en). Only the side files changed: `tokens.txt` became `vocab.txt`, and `config.json` carries the FunASR frontend statistics from `am.mvn`. ## Usage The `paraformer` model type is on the `feat/paraformer` branch of the TigreGotico [onnx-asr](https://github.com/TigreGotico/onnx-asr) fork. ```sh pip install "onnx-asr[cpu,hub] @ git+https://github.com/TigreGotico/onnx-asr@feat/paraformer" ``` ```py import onnx_asr model = onnx_asr.load_model("OpenVoiceOS/paraformer-trilingual-zh-yue-en-onnx") print(model.recognize("audio.wav")) ``` ## Graph contract | Item | Value | |---|---| | Input | `speech`, float32, `[batch, num_frames, 560]` | | Input | `speech_lengths`, int32, `[batch]` | | Output | `logits`, float32, `[batch, num_tokens, 11666]` | | Output | `token_num`, int32, `[batch]`, the CIF token count | | Output (unused) | `decoder_hidden`, `pre_acoustic_embeds` — SeACo hotword pathway, not read by onnx-asr | The 560 dim input is the FunASR frontend: an 80 dim kaldi fbank of a waveform scaled to the int16 range, then a low frame rate stack of 7 frames with a hop of 6, then the `am.mvn` mean variance statistics. onnx-asr computes the fbank with its `wespeaker` preprocessor and applies the LFR stack and the CMVN in the runtime. Decoding is one argmax per logits row, stopping at `` and never reading past `token_num`. ## Streaming A streaming Paraformer also exists upstream, using a different graph with encoder and decoder states, and needs a streaming runtime that onnx-asr does not have yet (upstream issue #21). Only the offline model is mirrored here. ## Files `model.onnx` (871 MB) and `model_int8.onnx` (245 MB). Both come straight from the sherpa-onnx repository; nothing was re-quantized here. ## Parity with native FunASR 2 Mandarin, 2 English and 2 Cantonese clips (FLEURS `cmn_hans_cn`, `en_us`, `yue_hant_hk`), native FunASR 1.4.0 on the exact ModelScope checkpoint (`dengcunqin/speech_seaco_paraformer_large_asr_nat-zh-cantonese-en-16k-common-vocab11666-pytorch`) with `frontend.dither = 0`. | Clip | fp32 | int8 | | --- | --- | --- | | zh_1 | identical | identical | | zh_2 | identical | identical | | en_1 | identical | identical | | en_2 | identical | a few words differ (deliberately out-of-domain foreign-cuisine clip) | | yue_1 | identical | identical | | yue_2 | identical | identical | 6 of 6 fp32 comparisons and 5 of 6 int8 comparisons are character identical to native FunASR. The one int8 difference is on the same deliberately hard, out-of-domain English clip that shows quantization noise in the other Paraformer mirrors in this collection. Cantonese, the language this checkpoint adds to the collection, is perfect in both precisions. ## Speed AMD Ryzen 5 7600, 6 cores, `OMP_NUM_THREADS=6`, CPU provider, batch 1, warm cache, best of 3 passes over the 6 test clips. | Precision | RTF | | --- | --- | | fp32 | 0.0196 | | int8 | 0.0148 | ## License Apache License 2.0. The [ModelScope model card](https://www.modelscope.cn/models/dengcunqin/speech_seaco_paraformer_large_asr_nat-zh-cantonese-en-16k-common-vocab11666-pytorch) states `License: Apache License 2.0`. The sherpa-onnx mirror carries no license file of its own, which is why the ModelScope source was read directly rather than the mirror. * Model: Alibaba DAMO Academy / FunASR, `dengcunqin/speech_seaco_paraformer_large_asr_nat-zh-cantonese-en-16k-common-vocab11666-pytorch`, Apache-2.0. * ONNX export: [sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx) by Fangjun Kuang (csukuangfj), Xiaomi. * This repository only repackages those files for onnx-asr. ## Deviations * The LFR stack and the CMVN are in the runtime class, not in the graph, matching every other Paraformer mirror in this collection. * No timestamps. Paraformer has no per-token frame index. * No inverse text normalization and no homophone replacement. * The streaming Paraformer is out of scope (see above). * This checkpoint is SeACo-Paraformer (hotword biasing), used here without hotwords. The bias-pathway outputs are present in the graph but unread.