lwittich/parakeet-tdt-0.6b-v3-onnx-int8-cpu

Parakeet TDT 0.6B v3, ONNX INT8

This repository provides an ONNX Runtime compatible, selectively INT8-quantized variant of NVIDIA's nvidia/parakeet-tdt-0.6b-v3 automatic speech recognition model.

The model was prepared for CPU-based inference under constrained local deployment conditions. The intended use case is low-latency multilingual ASR in local speech-processing pipelines where inference cost, memory use, and real-time behaviour are important deployment constraints.

This is not an official NVIDIA release.

Attribution

This model is derived from NVIDIA's Parakeet TDT 0.6B v3.

Item	Value
Original model	nvidia/parakeet-tdt-0.6b-v3
Original author	NVIDIA
Original license	CC-BY-4.0
Modification	ONNX export and selective dynamic INT8 quantization
Quantized operators	MatMul, Gemm
Excluded operators	Conv

Quickstart

pip install onnxruntime onnx-asr[cpu,hub]

import onnx_asr

model = onnx_asr.load_model(
    "<REPO_ID>",
    providers=["CPUExecutionProvider"],
)

text = model.recognize(audio_array)
print(text)

Input audio must be:

• mono
• 16 kHz
• float32 NumPy array

Quantization

Dynamic quantization was performed using ONNX Runtime. Quantization is applied selectively to matrix multiplication operators in order to reduce model size and CPU inference cost while preserving the original convolution operators.

Component	Quantization scope
Encoder	Dynamic INT8 weight quantization for MatMul and Gemm
Decoder / joint	Dynamic INT8 weight quantization for MatMul and Gemm
Activations	FP32
Convolution operators	FP32

Quantized operators

• MatMul
• Gemm

Excluded operators

• Conv

Rationale

Dynamic INT8 weight quantization reduces the storage and memory bandwidth requirements of matrix multiplication weights while keeping activations in FP32. Convolution operators were left unquantized because quantizing them increased runtime overhead and degraded transcription quality in the tested CPU execution environment.

Benchmark Summary

Variant	Corpus WER (%)	Median RTF	p95 RTF
ONNX INT8	6.8	0.031	0.039

Evaluation setup

Property	Value
Dataset	FLEURS test split
Languages	25 European languages supported by the base model
Execution provider	ONNX Runtime CPUExecutionProvider
CPU cores	4
Memory limit	8 GiB
Audio format	Mono, 16 kHz
Metric	Word error rate (WER)
Runtime metric	Real-time factor (RTF)

WER was computed after:

• Unicode NFKC normalization
• lowercasing
• punctuation removal
• whitespace normalization

RTF is computed as:

RTF = inference_time_seconds / audio_duration_seconds

Values below 1.0 indicate faster-than-real-time inference.

Repository structure

.
├── encoder-model.onnx
├── encoder-model.onnx.data
├── decoder_joint-model.onnx
├── decoder_joint-model.onnx.data
├── config.json
├── quantization_config.json
├── vocab.txt
└── README.md

Intended use

This model is intended for research and development involving local multilingual ASR, CPU inference, and constrained deployment studies. It may be useful as a component in cascaded speech-to-text and text-translation pipelines.

Limitations

• This is not an official NVIDIA release.
• The model is derived from the original NVIDIA Parakeet TDT 0.6B v3 model and inherits its language coverage and modelling limitations.
• The quantization is selective and weight-only. It should not be interpreted as full INT8 inference.
• Runtime performance depends on CPU architecture, ONNX Runtime version, threading configuration, and memory bandwidth.
• The benchmark results are specific to the stated evaluation setup and should not be assumed to generalize to all devices.
• The model performs ASR only. It does not perform speech translation or text translation by itself.

License

This repository follows the original model license: CC-BY-4.0.

The original model is © NVIDIA Corporation. This repository contains a converted and quantized derivative artifact.

Citation

@misc{wittich2026parakeet_tdt_onnx_int8,
  title        = {Parakeet TDT 0.6B v3 ONNX INT8},
  author       = {Wittich, Lucas},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/<REPO_ID>}},
  note         = {ONNX Runtime compatible selectively INT8-quantized derivative of NVIDIA Parakeet TDT 0.6B v3}
}