Parakeet TDT 0.6B v3 — weight-only int4 ONNX for sherpa-onnx
A 4-bit weight-only quantization of NVIDIA's Parakeet TDT 0.6B v3 for sherpa-onnx, built from the fp32 export csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3. It is the model shipped by OpenRamble for Windows, a local dictation app.
Why not the stock int8 export
The published int8 export is dynamically quantized (DynamicQuantizeLinear + MatMulInteger /
ConvInteger): activations are cut to 8 bits at run time, including in the convolutional front
end that sees the spectrogram first. On a quiet or muffled microphone — a laptop microphone array
peaking around −25 dBFS — it answered Russian speech with nothing or with English filler, while the
fp32 export read the same recording correctly.
This build quantizes only the encoder's matrix weights (MatMulNBits, 4-bit, symmetric, blocks of
64, accuracy_level=4); activations stay fp32. Decoder and joiner are the unmodified fp32 files.
Measured on FLEURS ru dev (40 utterances, 710 words), sherpa-onnx 1.13.5, CPU:
| variant | WER | CER | size |
|---|---|---|---|
| stock int8 | 7.3% | 2.2% | 640 MB |
| this (int4 WO) | 5.5% | 1.5% | 719 MB |
| fp32 | 5.4% | 1.4% | 2.4 GB |
Same speed as the int8 export; peak memory about the same.
Files
| file | what |
|---|---|
encoder.int4.onnx |
FastConformer encoder, MatMulNBits 4-bit weights, self-contained (no external data) |
decoder.onnx |
prediction network, fp32 (from the source repo) |
joiner.onnx |
joiner, fp32 (from the source repo) |
tokens.txt |
SentencePiece tokens |
Requires an ONNX Runtime with the MatMulNBits contrib op (sherpa-onnx ≥ 1.13 / ORT ≥ 1.17).
Usage (sherpa-onnx, Python)
import sherpa_onnx
rec = sherpa_onnx.OfflineRecognizer.from_transducer(
encoder="encoder.int4.onnx", decoder="decoder.onnx", joiner="joiner.onnx",
tokens="tokens.txt", model_type="nemo_transducer", num_threads=4)
Reproduce
scripts/quantize-parakeet-int4.py in the OpenRamble for Windows repository downloads the fp32
export and produces exactly these files (deterministic; SHA-256 of the encoder is stable across
runs).
License
CC-BY-4.0, as the original model. Credit: NVIDIA (model), k2-fsa / csukuangfj (ONNX export).
- Downloads last month
- -
Model tree for hmepas/parakeet-tdt-0.6b-v3-int4
Base model
nvidia/parakeet-tdt-0.6b-v3
import nemo.collections.asr as nemo_asr asr_model = nemo_asr.models.ASRModel.from_pretrained("hmepas/parakeet-tdt-0.6b-v3-int4") transcriptions = asr_model.transcribe(["file.wav"])