hmepas/parakeet-tdt-0.6b-v3-int4

Parakeet TDT 0.6B v3 — weight-only int4 ONNX for sherpa-onnx

A 4-bit weight-only quantization of NVIDIA's Parakeet TDT 0.6B v3 for sherpa-onnx, built from the fp32 export csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3. It is the model shipped by OpenRamble for Windows, a local dictation app.

Why not the stock int8 export

The published int8 export is dynamically quantized (DynamicQuantizeLinear + MatMulInteger / ConvInteger): activations are cut to 8 bits at run time, including in the convolutional front end that sees the spectrogram first. On a quiet or muffled microphone — a laptop microphone array peaking around −25 dBFS — it answered Russian speech with nothing or with English filler, while the fp32 export read the same recording correctly.

This build quantizes only the encoder's matrix weights (MatMulNBits, 4-bit, symmetric, blocks of 64, accuracy_level=4); activations stay fp32. Decoder and joiner are the unmodified fp32 files.

Measured on FLEURS ru dev (40 utterances, 710 words), sherpa-onnx 1.13.5, CPU:

variant	WER	CER	size
stock int8	7.3%	2.2%	640 MB
this (int4 WO)	5.5%	1.5%	719 MB
fp32	5.4%	1.4%	2.4 GB

Same speed as the int8 export; peak memory about the same.

Files

file	what
encoder.int4.onnx	FastConformer encoder, MatMulNBits 4-bit weights, self-contained (no external data)
decoder.onnx	prediction network, fp32 (from the source repo)
joiner.onnx	joiner, fp32 (from the source repo)
tokens.txt	SentencePiece tokens

Requires an ONNX Runtime with the MatMulNBits contrib op (sherpa-onnx ≥ 1.13 / ORT ≥ 1.17).

Usage (sherpa-onnx, Python)

import sherpa_onnx
rec = sherpa_onnx.OfflineRecognizer.from_transducer(
    encoder="encoder.int4.onnx", decoder="decoder.onnx", joiner="joiner.onnx",
    tokens="tokens.txt", model_type="nemo_transducer", num_threads=4)

Reproduce

scripts/quantize-parakeet-int4.py in the OpenRamble for Windows repository downloads the fp32 export and produces exactly these files (deterministic; SHA-256 of the encoder is stable across runs).

License

CC-BY-4.0, as the original model. Credit: NVIDIA (model), k2-fsa / csukuangfj (ONNX export).