Abdullahu5mani/granite-speech-4.1-2b-nar-portable

Granite Speech 4.1 2B NAR Portable INT4 for Taurscribe

This repository contains a portable ONNX Runtime artifact derived from IBM's Granite Speech 4.1 2B NAR model for local transcription in Taurscribe.

It is designed for Windows systems without an NVIDIA CUDA GPU. It supports:

• full GPU inference through ONNX Runtime DirectML on compatible AMD, Intel, and NVIDIA DirectX 12 GPUs;
• multi-threaded ONNX Runtime CPU inference as a portable fallback.

This is not a newly trained or fine-tuned model. The learned Granite weights, tokenizer, vocabulary, and supported languages are inherited from IBM's base model. Taurscribe changed the ONNX representation and host runtime path.

This is a multi-graph Taurscribe runtime bundle, not a single-file Transformers replacement. Correct inference requires Taurscribe's feature extraction, CTC, projection, edit-slot, and tokenizer pipeline.

What Changed

The portable artifact builds on Taurscribe's INT4/token-ID Granite bundle:

1. Selected large MatMul weights use INT4 weight-only quantization through ONNX MatMulNBits.
2. editor.onnx performs ArgMax in the graph and returns token_ids instead of transferring the full sequence-by-vocabulary logits tensor to Rust.
3. Thirty-two rank-5 windowed-attention MatMul operations in encoder.onnx are reshaped to equivalent rank-3 batched operations supported by DirectML.
4. Runtime shape chains are evaluated for Taurscribe's fixed [1, 800, 160] encoder bucket, stored as constants, and approximately 950 dead shape nodes are removed.
5. Sixteen two-output GLU Split operations are replaced with equivalent Slice pairs to avoid incorrect output inside DirectML fused partitions.

Only encoder.onnx receives the DirectML compatibility rewrites. The projector, token embedding graph, editor, tokenizer, and model weights are copied from the INT4/token-ID parent artifact.

Runtime Placement

Full DirectML:

encoder.onnx      -> DirectML
projector.onnx    -> DirectML
embed_tokens.onnx -> DirectML
editor.onnx       -> DirectML

CPU fallback:

encoder.onnx      -> CPU
projector.onnx    -> CPU
embed_tokens.onnx -> CPU
editor.onnx       -> CPU

The ONNX files do not select a GPU vendor. Taurscribe's Rust code asks ONNX Runtime to create DirectML or CPU sessions. See taurscribe_granite_nar_manifest.json for the graph contract.

Validation

The conversion script compares every rewritten encoder output against the source encoder on CPU and aborts if the worst relative difference exceeds 1e-3. The produced artifact measured a worst relative difference of about 4.4e-5. DirectML encoder BPE argmax agreement with CPU was 1.0000 on the validated input.

The following end-to-end Taurscribe results use the same 30 utterances from LibriSpeech test-clean (mean processed duration: 8.47 seconds):

Route	Hardware	Mean latency	Mean RTF	Mean WER
CUDA reference artifact	RTX 4070 Laptop	0.250 s	0.040	4.31%
This artifact, full DirectML	Radeon 780M	4.045 s	0.643	4.31%
This artifact, CPU (8 threads)	Ryzen 7 8845HS	8.823 s	1.415	4.31%
Historical hybrid baseline	1-thread CPU encoder + DirectML	13.135 s	2.194	4.31%

These are limited local validation results, not a replacement for the base model's full published evaluation. See BENCHMARKS.md for methodology and tail latency.

Hardware Compatibility

• AMD Radeon 780M DirectML: validated end to end with CPU fallback disabled.
• NVIDIA DirectML: graph-level compatibility validated; use the separate CUDA artifact for substantially better NVIDIA performance.
• Intel DirectML: expected to work on DirectX 12 hardware but not yet validated with this exact artifact.
• x86-64 CPU: supported through ONNX Runtime CPU; actual speed depends heavily on memory bandwidth and thread count.

DirectML requires Windows and a DirectX 12-compatible GPU. A GPU driver or execution-provider failure should be handled by the host application with CPU fallback. For production hardware certification, run the DirectML probe in a separate helper process so a native driver crash cannot terminate the app.

Taurscribe Selection

Taurscribe model ID:

granite-speech-4.1-2b-nar-portable

Development overrides:

$env:TAURSCRIBE_GRANITE_BACKEND = "directml"
$env:TAURSCRIBE_GRANITE_DML_DEVICE_ID = "0"
$env:TAURSCRIBE_GRANITE_BACKEND = "cpu"
$env:TAURSCRIBE_GRANITE_CPU_THREADS = "8"

The DirectML device ID follows DXGI adapter order and is machine-specific.

Files

encoder.onnx
encoder.onnx.data
projector.onnx
projector.onnx.data
embed_tokens.onnx
editor.onnx
editor.onnx.data
tokenizer.json
tokenizer_config.json
preprocessor_config.json
processor_config.json
generation_config.json
taurscribe_granite_nar_manifest.json
manifest.json
BENCHMARKS.md
NOTICE
LICENSE

manifest.json records artifact metadata, exact file sizes, and SHA-256 checksums for the runtime payload.

Source and Attribution

• Base model: ibm-granite/granite-speech-4.1-2b-nar
• Original provider: IBM Granite Team
• Original license: Apache License 2.0
• ONNX optimization and packaging: Taurscribe project

No additional training or fine-tuning was performed by Taurscribe.