# Hybrid Naming Scheme & Benchmark Synopsis

This report summarizes baseline and hybrid quantization results for `granite-4.0-h-350m-unsloth` as measured by the Magic Quant pipeline.

## Naming Scheme

Model variants follow a structured suffix convention that encodes both the base conversion mode and per-tensor quantization schemes.

| Suffix Example | Meaning |
| -------------- | ------- |
| `BF16` | Pure full-precision family baseline (no quantization). |
| `Q8_0`, `Q6_K`, `Q5_K`, `Q4_K_M`, `IQ4_NL`, `MXFP4_MOE` | Pure model-wide quantization baselines. |
| `iq4_nl-emb_Q4_K-head_Q4_K-moe_rt_Q4_K` | Base conversion mode `iq4_nl` with per-group schemes: embeddings (`emb_`), output head (`head_`), MoE router (`moe_rt_`). |
| `...-aq_F16-akv_Q8_0-fd_Q4_K-ao_Q5_K` | Extended sensitivity groups: Attention Q (`aq_`), Attention K+V (`akv_`), FFN Down (`fd_`), Attention Output (`ao_`). |
| `mxfp4_moe-emb_IQ4_NL-head_Q6_K-moe_exp_MXFP4-moe_rt_Q6_K` | MXFP4-centric hybrids with MoE expert group (`moe_exp_`) and mixed IQ / Q-schemes per tensor group. |

In general, anything after the base model name is a purely mechanical description of **how** the weights were transformed, not a new training run.

---

## Benchmark Methodology

All models were tested with a unified automated harness using `llama.cpp` tools.

**Included tests:**

- **Throughput:**  
    `llama-bench` with descending GPU offload (`-ngl 35 → 0`) and automatic OOM retry.  
    Highest successful TPS is recorded.

- **Perplexity:**  
    Three domains: **general**, **code**, **math**.  
    Each uses an auto-generated corpus of ~**32k tokens**.  
    Perplexity is computed with `llama-perplexity` at **2048-token** context.  
    Same GPU retry logic as above.

- **Precision loss:**  
    Each model is compared to its **family BF16 baseline**.  
    Precision-loss % is computed for all PPL domains, plus an averaged score.  
    Models are ranked by this metric.

---

### Table - Overview of Results

Comparing to BF16.

| model_name | size_reduction | tps_change |
| ---------- | -------------- | ---------- |
| mxfp4_moe-akv_BF16-ao_Q6_K-aq_Q8_0-emb_BF16-fd_BF16-fug_BF16 | 15.62% | -0.75% |
| mxfp4_moe-akv_Q8_0-ao_Q6_K-aq_Q8_0-emb_Q8_0-fd_Q8_0-fug_Q8_0 | 46.88% | -6.54% |
| mxfp4_moe-akv_Q8_0-ao_Q5_K-aq_Q6_K-emb_Q8_0-fd_Q8_0-fug_Q8_0 | 46.88% | -5.34% |
| mxfp4_moe-akv_Q6_K-ao_Q6_K-aq_Q6_K-emb_Q8_0-fd_Q8_0-fug_Q8_0 | 46.88% | -4.43% |
| Q5_K | 62.50% | -24.03% |

* All percentages compared against the selected family BF16 baseline.

---

### Table - File Size + TPS + Avg Precision Loss

| model_name | file_size_gb | bench_tps | avg_prec_loss |
| ---------- | ------------ | --------- | ------------- |
| BF16 | 0.64 | 1718.28 | 0.0000 |
| mxfp4_moe-akv_BF16-ao_Q6_K-aq_Q8_0-emb_BF16-fd_BF16-fug_BF16 | 0.54 | 1705.35 | 0.0816 |
| mxfp4_moe-akv_Q8_0-ao_Q6_K-aq_Q8_0-emb_Q8_0-fd_Q8_0-fug_Q8_0 | 0.34 | 1605.97 | 0.2555 |
| mxfp4_moe-akv_Q8_0-ao_Q5_K-aq_Q6_K-emb_Q8_0-fd_Q8_0-fug_Q8_0 | 0.34 | 1626.49 | 0.2776 |
| mxfp4_moe-akv_Q6_K-ao_Q6_K-aq_Q6_K-emb_Q8_0-fd_Q8_0-fug_Q8_0 | 0.34 | 1642.13 | 0.3407 |
| Q5_K | 0.24 | 1305.37 | 2.8875 |

* `avg_prec_loss` is the averaged absolute precision-loss % vs BF16.

---

### Table - PPL Columns

| model_name | gen | gen_er | code | code_er | math | math_er |
| ---------- | --- | ------ | ---- | ------- | ---- | ------- |
| BF16 | 18.1312 | 0.4655 | 1.9549 | 0.0175 | 10.2880 | 0.2315 |
| mxfp4_moe-akv_BF16-ao_Q6_K-aq_Q8_0-emb_BF16-fd_BF16-fug_BF16 | 18.1560 | 0.4667 | 1.9548 | 0.0175 | 10.2986 | 0.2319 |
| mxfp4_moe-akv_Q8_0-ao_Q6_K-aq_Q8_0-emb_Q8_0-fd_Q8_0-fug_Q8_0 | 18.2304 | 0.4691 | 1.9555 | 0.0175 | 10.3074 | 0.2320 |
| mxfp4_moe-akv_Q8_0-ao_Q5_K-aq_Q6_K-emb_Q8_0-fd_Q8_0-fug_Q8_0 | 18.2468 | 0.4695 | 1.9583 | 0.0176 | 10.2902 | 0.2312 |
| mxfp4_moe-akv_Q6_K-ao_Q6_K-aq_Q6_K-emb_Q8_0-fd_Q8_0-fug_Q8_0 | 18.2573 | 0.4696 | 1.9580 | 0.0175 | 10.3053 | 0.2318 |
| Q5_K | 18.9974 | 0.4899 | 1.9842 | 0.0180 | 10.5335 | 0.2365 |

* gen = ppl_general, code = ppl_code, math = ppl_math

---

### Table - Precision Loss Columns

| model_name | loss_general | loss_code | loss_math |
| ---------- | ------------ | --------- | --------- |
| BF16 | 0.0000 | 0.0000 | 0.0000 |
| mxfp4_moe-akv_BF16-ao_Q6_K-aq_Q8_0-emb_BF16-fd_BF16-fug_BF16 | 0.1368 | 0.0051 | 0.1030 |
| mxfp4_moe-akv_Q8_0-ao_Q6_K-aq_Q8_0-emb_Q8_0-fd_Q8_0-fug_Q8_0 | 0.5471 | 0.0307 | 0.1886 |
| mxfp4_moe-akv_Q8_0-ao_Q5_K-aq_Q6_K-emb_Q8_0-fd_Q8_0-fug_Q8_0 | 0.6376 | 0.1739 | 0.0214 |
| mxfp4_moe-akv_Q6_K-ao_Q6_K-aq_Q6_K-emb_Q8_0-fd_Q8_0-fug_Q8_0 | 0.6955 | 0.1586 | 0.1682 |
| Q5_K | 4.7774 | 1.4988 | 2.3863 |

* loss_* values are absolute precision-loss % vs BF16 per domain.