qwertz92/ibm-granite-speech-4.1-2b-nar-q4-rtn-onnx

Granite Speech 4.1 2B NAR — q4-RTN (INT4) ONNX

Read first: INT8 is the better choice for most people

This is a 4-bit (q4/INT4) weight-only ONNX export of the Granite Speech 4.1 2B NAR editor using RTN (round-to-nearest) quantisation. It is published for transparency and research, not because it is faster. On a desktop CPU (tested: Ryzen 5 7600X) it is slower than the INT8 export and only about 9-16 % smaller — not half the size. Its only theoretical advantage (4-bit = less memory bandwidth) needs a GPU, and the NAR model cannot currently run its encoder on the GPU (DirectML/WebGPU cannot run the conformer encoder ops), so the q4 advantage does not materialise end-to-end on consumer hardware today.

Use smcleod/ibm-granite-speech-4.1-2b-nar-onnx INT8 instead unless you specifically need 4-bit weights for research or a runtime/accelerator where INT4 GEMM wins. HQQ was also tested locally but is intentionally not the primary published artifact: it is larger/slower and is numerically broken on DirectML's asymmetric MatMulNBits path.

We did the work and measured it so you don't have to repeat it.

What this is

A mixed-precision bundle, q4 only where 4-bit actually helps:

• editor.onnx — the 2B "editor" LLM, quantised to 4-bit RTN (com.microsoft.MatMulNBits).
• encoder.onnx — the conformer + CTC head, kept INT8 (a q4 encoder is larger — its Convs stay FP32 — and noisier).
• embed_tokens.onnx — kept fp16w (a Gather; nothing to 4-bit-quantise).

The file layout mirrors the upstream ONNX export, with the mixed-precision graph set under q4/.

Benchmarks (Ryzen 5 7600X / Intel Arc A750)

Variant	Bundle	CPU RTF (lower=faster)	DirectML	Quality
INT8 (reference)	2.32 GiB	0.53–0.60	encoder fails	English verbatim-correct
q4-RTN	1.96 GiB	0.62	editor ok, no end-to-end win	English verbatim-correct
q4-HQQ (tested locally, not published here)	2.12 GiB	0.70	broken	correct on CPU only

Why INT8 wins on CPU: native INT8 GEMM (AVX-512 VNNI) beats q4's per-block dequantisation overhead; 4-bit only pays off when memory-bandwidth-bound (GPU). Why GPU doesn't help: profiling shows the conformer encoder is ~90 % of the runtime and is CPU-locked (its 5-D batched attention MatMul isn't supported by the DirectML EP; WebGPU hits an Einsum shader bug). The editor runs on DirectML (~2–3× faster than INT8-CPU in isolation at realistic sizes), but at ~10 % of the total it can't move the end-to-end number, and the GPU→CPU transfer of the 100k-vocab logits makes the hybrid slightly slower.

Shape inspection found 32 high-rank encoder attention MatMul nodes plus 16 Einsum nodes, so a GPU-capable NAR path needs a real encoder re-export or graph rewrite, not a small runtime switch.

Provenance & how it was made

• Base model: ibm-granite/granite-speech-4.1-2b-nar (IBM, Apache-2.0).
• ONNX source graphs: smcleod/ibm-granite-speech-4.1-2b-nar-onnx (Apache-2.0).
• q4: onnxruntime MatMulNBitsQuantizer over smcleod's FP32 editor, block size 32, QOperator/MatMulNBits. This repository publishes RTN = DefaultWeightOnlyQuantConfig. HQQ = HQQWeightOnlyQuantConfig was evaluated locally and documented, but not used as the public artifact.

Inference note (NAR decode)

The encoder BPE/CTC head emits vocab+1 = 100353 classes with the blank prepended at index 0; a non-blank class c is LLM token c-1. CTC decode: per-frame argmax → collapse consecutive → drop blank 0 → subtract 1; feed the ids directly into the editor's eos-filled insertion slots (do not decode to text and re-encode). (config.blank_token_id=100257 is the editor/slot blank, not the CTC blank.)

Licence

Apache-2.0, inheriting the upstream IBM model and the smcleod ONNX export.