valoomba/nemotron-speech-en-0.6b-onnx-1120ms-int4

nemotron-speech-en-0.6b — ONNX 1120ms, int4 (WebGPU)

A 4-bit (MatMulNBits) quantization of the FP16 1120 ms ONNX export of NVIDIA's nvidia/nemotron-speech-streaming-en-0.6b (English FastConformer encoder + RNN-T transducer, with built-in punctuation & capitalization), for running on-device in the browser via onnxruntime-web + WebGPU.

• ~469 MB total (encoder ~451 MB int4 + fused decoder ~18 MB fp16) vs ~1.25 GB for the FP16 build.
• Quantized only the encoder's MatMul weights to int4 MatMulNBits (217 ops); convs and the small fused prediction+joint decoder stay higher precision. No ConvInteger, so it loads and runs on the onnxruntime-web WebGPU execution provider.
• Accuracy matches the FP16 export on tested clips (proper nouns, punctuation preserved).

Provenance

• Base model: nvidia/nemotron-speech-streaming-en-0.6b (see LICENSE.txt / NOTICE.txt).
• FP16 1120 ms ONNX export: shameez/nemotron-speech-streaming-en-0.6b-onnx-1120ms.
• int4 quantization: onnxruntime.quantization.matmul_nbits_quantizer (bits=4, block_size=128) — see quantize.py.

Files

file	precision	notes
encoder_model.onnx (+.data)	int4	cache-aware streaming FastConformer encoder
decoder_model.onnx (+.data)	fp16	fused RNN-T prediction net + joint
tokens.txt	—	SentencePiece pieces (<piece> <id>), vocab 1024 + blank 1024

I/O (unchanged from the FP16 export)

• Encoder: audio_signal[1,128,121] (112 new + 9 overlap mel frames, log-mel 128/16kHz), length i64, batch-first caches cache_last_channel[1,24,70,1024] / cache_last_time[1,24,1024,8], cache_last_channel_len i64 → outputs[1,1024,14] (channel-major) + *_next caches.
• Decoder (fused): encoder_outputs[1,1024,1], targets i32, target_length i32, input_states_1/2[2,1,640] → outputs[…,1025] logits, output_states_1/2. blank id = 1024.