File size: 5,831 Bytes

77d19a8

---
license: apache-2.0
base_model: ibm-granite/granite-speech-4.1-2b-nar
library_name: onnx
pipeline_tag: automatic-speech-recognition
language:
  - en
  - fr
  - de
  - es
  - pt
tags:
  - automatic-speech-recognition
  - granite-speech
  - onnx
  - onnxruntime
  - directml
  - cpu
  - int4
  - matmulnbits
  - taurscribe
---

# Granite Speech 4.1 2B NAR Portable INT4 for Taurscribe

This repository contains a portable ONNX Runtime artifact derived from IBM's
[Granite Speech 4.1 2B NAR](https://huggingface.co/ibm-granite/granite-speech-4.1-2b-nar)
model for local transcription in Taurscribe.

It is designed for Windows systems without an NVIDIA CUDA GPU. It supports:

- full GPU inference through ONNX Runtime DirectML on compatible AMD, Intel,
  and NVIDIA DirectX 12 GPUs;
- multi-threaded ONNX Runtime CPU inference as a portable fallback.

This is not a newly trained or fine-tuned model. The learned Granite weights,
tokenizer, vocabulary, and supported languages are inherited from IBM's base
model. Taurscribe changed the ONNX representation and host runtime path.

> This is a multi-graph Taurscribe runtime bundle, not a single-file
> Transformers replacement. Correct inference requires Taurscribe's feature
> extraction, CTC, projection, edit-slot, and tokenizer pipeline.

## What Changed

The portable artifact builds on Taurscribe's INT4/token-ID Granite bundle:

1. Selected large `MatMul` weights use INT4 weight-only quantization through
   ONNX `MatMulNBits`.
2. `editor.onnx` performs `ArgMax` in the graph and returns `token_ids` instead
   of transferring the full sequence-by-vocabulary logits tensor to Rust.
3. Thirty-two rank-5 windowed-attention `MatMul` operations in `encoder.onnx`
   are reshaped to equivalent rank-3 batched operations supported by DirectML.
4. Runtime shape chains are evaluated for Taurscribe's fixed `[1, 800, 160]`
   encoder bucket, stored as constants, and approximately 950 dead shape nodes
   are removed.
5. Sixteen two-output GLU `Split` operations are replaced with equivalent
   `Slice` pairs to avoid incorrect output inside DirectML fused partitions.

Only `encoder.onnx` receives the DirectML compatibility rewrites. The
projector, token embedding graph, editor, tokenizer, and model weights are
copied from the INT4/token-ID parent artifact.

## Runtime Placement

Full DirectML:

```text
encoder.onnx      -> DirectML
projector.onnx    -> DirectML
embed_tokens.onnx -> DirectML
editor.onnx       -> DirectML
```

CPU fallback:

```text
encoder.onnx      -> CPU
projector.onnx    -> CPU
embed_tokens.onnx -> CPU
editor.onnx       -> CPU
```

The ONNX files do not select a GPU vendor. Taurscribe's Rust code asks ONNX
Runtime to create DirectML or CPU sessions. See
`taurscribe_granite_nar_manifest.json` for the graph contract.

## Validation

The conversion script compares every rewritten encoder output against the
source encoder on CPU and aborts if the worst relative difference exceeds
`1e-3`. The produced artifact measured a worst relative difference of about
`4.4e-5`. DirectML encoder BPE argmax agreement with CPU was `1.0000` on the
validated input.

The following end-to-end Taurscribe results use the same 30 utterances from
LibriSpeech `test-clean` (mean processed duration: 8.47 seconds):

| Route | Hardware | Mean latency | Mean RTF | Mean WER |
|---|---|---:|---:|---:|
| CUDA reference artifact | RTX 4070 Laptop | 0.250 s | 0.040 | 4.31% |
| **This artifact, full DirectML** | **Radeon 780M** | **4.045 s** | **0.643** | **4.31%** |
| **This artifact, CPU (8 threads)** | **Ryzen 7 8845HS** | **8.823 s** | **1.415** | **4.31%** |
| Historical hybrid baseline | 1-thread CPU encoder + DirectML | 13.135 s | 2.194 | 4.31% |

These are limited local validation results, not a replacement for the base
model's full published evaluation. See `BENCHMARKS.md` for methodology and tail
latency.

## Hardware Compatibility

- **AMD Radeon 780M DirectML:** validated end to end with CPU fallback disabled.
- **NVIDIA DirectML:** graph-level compatibility validated; use the separate
  CUDA artifact for substantially better NVIDIA performance.
- **Intel DirectML:** expected to work on DirectX 12 hardware but not yet
  validated with this exact artifact.
- **x86-64 CPU:** supported through ONNX Runtime CPU; actual speed depends
  heavily on memory bandwidth and thread count.

DirectML requires Windows and a DirectX 12-compatible GPU. A GPU driver or
execution-provider failure should be handled by the host application with CPU
fallback. For production hardware certification, run the DirectML probe in a
separate helper process so a native driver crash cannot terminate the app.

## Taurscribe Selection

Taurscribe model ID:

```text
granite-speech-4.1-2b-nar-portable
```

Development overrides:

```powershell
$env:TAURSCRIBE_GRANITE_BACKEND = "directml"
$env:TAURSCRIBE_GRANITE_DML_DEVICE_ID = "0"
$env:TAURSCRIBE_GRANITE_BACKEND = "cpu"
$env:TAURSCRIBE_GRANITE_CPU_THREADS = "8"
```

The DirectML device ID follows DXGI adapter order and is machine-specific.

## Files

```text
encoder.onnx
encoder.onnx.data
projector.onnx
projector.onnx.data
embed_tokens.onnx
editor.onnx
editor.onnx.data
tokenizer.json
tokenizer_config.json
preprocessor_config.json
processor_config.json
generation_config.json
taurscribe_granite_nar_manifest.json
manifest.json
BENCHMARKS.md
NOTICE
LICENSE
```

`manifest.json` records artifact metadata, exact file sizes, and SHA-256
checksums for the runtime payload.

## Source and Attribution

- Base model: [ibm-granite/granite-speech-4.1-2b-nar](https://huggingface.co/ibm-granite/granite-speech-4.1-2b-nar)
- Original provider: IBM Granite Team
- Original license: Apache License 2.0
- ONNX optimization and packaging: Taurscribe project

No additional training or fine-tuning was performed by Taurscribe.