---
license: apache-2.0
base_model: ibm-granite/granite-4.0-h-tiny
base_model_relation: quantized
pipeline_tag: text-generation
library_name: gguf
tags:
  - gguf
  - moe
  - apex
  - quantized
  - granite
  - mamba
  - hybrid
  - llama.cpp
---

# Granite-4.0-H-Tiny — APEX GGUF

MoE-aware, mixed-precision **APEX** quantization of
[ibm-granite/granite-4.0-h-tiny](https://huggingface.co/ibm-granite/granite-4.0-h-tiny)
— IBM's **hybrid Mamba-2 / Transformer MoE** (`granitemoehybrid`): 40 layers = 36
Mamba-2 + 4 attention, a 64-routed + shared-expert MoE FFN on every layer, ~7B
total / ~1B active, Apache-2.0.

To my knowledge this is the first APEX quant of a Granite hybrid Mamba-2/MoE.
APEX assigns precision per tensor role and per layer; here that meant teaching the
recipe about the Mamba-2 mixer tensors the stock generator doesn't know (see *Method*).

> **Quality gate (retrofit, added after initial publish):** real-generation coherence check
> passed on both shipped quants, plus a 3-turn chained tool-calling test (a distractor tool offered
> but correctly never called) — **3/3** on both. PPL-ratio vs bf16 also re-verified independently.
> See [GATE.md](GATE.md) and [TOOL_CALLING.md](TOOL_CALLING.md) for the full results and a
> reproducible transcript.

## Results

Perplexity on wikitext-2-raw (test, 200×512-token windows), `llama-perplexity`.

| File | Size | BPW | PPL | Δ vs bf16 |
|------|------|-----|-----|-----------|
| bf16 (reference) | 13 GB | 16.0 | 8.868 | — |
| **APEX i-quality** (ssm@Q6) | **4.4 GB** | 5.40 | **8.901** | **+0.38%** |
| **APEX i-quality — torch-imatrix** | **4.4 GB** | 5.40 | **8.864** | **−0.04%** |
| APEX hand-roll (ssm@Q8) | 4.5 GB | 5.54 | 8.913 | +0.51% |

The **torch-imatrix** row is the *same* APEX i-quality recipe, quantized with an
importance matrix computed independently in PyTorch (see [*The torch-imatrix
variant*](#the-torch-imatrix-variant)) instead of `llama-imatrix`. Its PPL (8.864)
is inside the ±0.11 error bar of both bf16 and the standard i-quality — i.e. the
two imatrices are interchangeable in quality.

Within **0.38% of full-precision perplexity at ~3× smaller**, and it runs comfortably
on modest hardware (~14 GB bf16 → 4.4 GB). Coherent, ~110 tok/s on a single GPU.
Built with a diverse imatrix (Bartowski `calibration_datav3`), full expert coverage.

## Usage (llama.cpp)
```bash
llama-cli   -m granite-4.0-h-tiny-APEX-i-quality.gguf -ngl 999 -p "Hello"
llama-server -m granite-4.0-h-tiny-APEX-i-quality.gguf -ngl 999 --host 0.0.0.0 --port 8080
```
Requires a llama.cpp build supporting the `granitemoehybrid` (a.k.a. `granitehybrid`)
architecture.

## Method
APEX is a bit-allocation recipe over stock `llama-quantize --tensor-type-file`.
Granite-H needed the **Mamba-2 mixer** tensors added to the map, which the stock
APEX generator omits:
- **Mamba-2** (36 layers): `ssm_in`, `ssm_conv1d`, `ssm_out` at mixer precision
  (Q6_K); 1-D state (`ssm_a`, `ssm_d`, `ssm_dt`, `ssm_norm`) left F32.
- **Attention** (4 layers): standard `attn_q/k/v/output` (Q6_K).
- **MoE** (all 40 layers): routed `ffn_*_exps` on a layer-depth precision gradient
  (Q6_K/Q5_K/IQ4_XS), **shared experts `ffn_*_shexp` at Q8_0** (always active → protect),
  router left high. Routed intermediate dim 512 is 256-divisible → no IQ4_NL workaround.

Config generation + patcher: `configs/`, `patch_granite_config.py`, `REPRODUCE.md`.
Baseline: IBM's own bf16 GGUF.

### Which file to pick, and a null result
**Use `i-quality`** — smaller *and* better. The `hand-roll` tier (also shipped) pins
the Mamba-2 recurrence tensors (`ssm_in/out/conv1d`) to Q8_0 to test whether protecting
the linear-recurrence state helps: it doesn't — it's **larger and slightly worse**
(4.5 GB, PPL 8.913 vs 8.901). Same null result seen on Kimi-Linear's KDA, so across two
hybrid architectures the SSM/recurrence tensors just aren't precision-sensitive. The
hand-roll is included to document the experiment; `i-quality` is the recommended tier.

## The torch-imatrix variant

`granite-4.0-h-tiny-APEX-i-quality-torch.gguf` (+ `granite-4.0-h-tiny-torch.imatrix`)
is a companion build that answers one question: **can the calibration imatrix be
produced without llama.cpp, and does that change the result?**

**What it is.** An importance matrix (imatrix) records, per weight tensor, the
per-input-channel sum of squared activations over calibration text — it tells
`llama-quantize` where to spend bits. Normally you get it from `llama-imatrix`,
which needs the whole model resident in RAM+VRAM. Here it was instead computed
**directly from the Hugging Face model in PyTorch**, by registering forward hooks
on every matmul (attention, Mamba-2 in/out projections, router, shared experts,
and per-expert routed FFNs) and accumulating Σx² as calibration text streams
through. The generator is **band-serialized**: it loads a few decoder layers at a
time, runs all calibration chunks through them, caches activations, frees them,
and moves on — so peak VRAM is a few layers, **not the whole model** (this run
peaked at ~6.4 GB on a 16 GB card, vs the ~14 GB a full-resident forward needs).
That decouples imatrix generation from llama.cpp's memory model: you can calibrate
a model far larger than your GPU by streaming it in bands.

**Everything else is identical.** Same bf16 base, same APEX `--tensor-type-file`
recipe, same `calibration_datav3`, same 126×512 calibration windows. The *only*
difference from the standard `i-quality` file is which tool produced the imatrix.
The two quants therefore differ **only** in the tensors whose quantization
consumes an imatrix (the K-/I-quant expert, attention and Mamba weights); the
Q8_0 shared experts and F32 tensors are bit-identical.

**Validation.** The PyTorch imatrix matches `llama-imatrix`'s output tensor-for-
tensor (368/368 names) with **median per-tensor correlation 0.995**. The handful
of lower-correlation tensors are the post-nonlinearity inputs (`ffn_down_shexp`,
`attn_output`) where transformers' naive Mamba scan and SiLU-gated intermediates
differ numerically from llama.cpp's kernels — a known wrinkle that does not affect
quality, since the quantizer only needs *relative* per-channel importance. The
proof is in the table above: the torch-imatrix quant scores **PPL 8.864**, inside
the ±0.11 error bar of both bf16 and the standard i-quality. The two imatrices are
interchangeable.

**Pick whichever you like** — they are equivalent in quality. The standard
`i-quality` is the reference; the `torch` build is provided for anyone who wants a
llama.cpp-free, VRAM-bounded path to the same result (e.g. calibrating very large
MoEs on modest hardware).

## Attribution & licenses
See [`LICENSE`](LICENSE) (Apache-2.0) and [`NOTICE`](NOTICE).
- Base: **IBM** ([@ibm-granite](https://huggingface.co/ibm-granite)) — [granite-4.0-h-tiny](https://huggingface.co/ibm-granite/granite-4.0-h-tiny) (Apache-2.0)
- Engine: **llama.cpp** ([@ggml-org](https://huggingface.co/ggml-org)) (MIT)
- APEX: **Ettore Di Giacinto / LocalAI** ([@mudler](https://huggingface.co/mudler)) — [localai-org/apex-quant](https://github.com/localai-org/apex-quant) (MIT)
- Calibration: **Bartowski** ([@bartowski](https://huggingface.co/bartowski)) — calibration_datav3

Unofficial community quantization; not affiliated with or endorsed by IBM.