# MagicQuant GGUF Hybrids - granite 4.0 h 350m unsloth

> **MagicQuant is an automated quantization, benchmarking, and evolutionary hybrid-GGUF search system for LLMs.**

Each release includes models optimized to outperform standard baseline quants (Q8, Q6, Q5, Q4).
If a baseline GGUF exists in this repo, the evolutionary engine couldn’t beat it.
If a baseline is missing, it’s because a hybrid configuration outperformed it so completely that including the baseline would've been pointless.

These hybrid GGUFs are built to be as small, fast, and low-drift as possible while preserving model capability.

To dive deeper into how MagicQuant works, see the main repo:
[MagicQuant on GitHub (by MagicCodingMan)](https://github.com/magiccodingman)

### Table - File Size + TPS + Avg Precision Loss 

| model_name | file_size_gb | bench_tps | avg_prec_loss |
| ---------- | ------------ | --------- | ------------- |
| [mxfp4_moe-EKUD:B16-O:Q6K-Q:Q8_0](./../../resolve/main/granite-4.0-h-350m-unsloth-mxfp4_moe-EKUD:B16-O:Q6K-Q:Q8_0.gguf?download=true) | 0.54 | 1705.35 | 0.0816 |
| [mxfp4_moe-O:Q6K-EQKUD:Q8_0](./../../resolve/main/granite-4.0-h-350m-unsloth-mxfp4_moe-O:Q6K-EQKUD:Q8_0.gguf?download=true) | 0.34 | 1605.97 | 0.2555 |
| [mxfp4_moe-O:Q5K-Q:Q6K-EKUD:Q8_0](./../../resolve/main/granite-4.0-h-350m-unsloth-mxfp4_moe-O:Q5K-Q:Q6K-EKUD:Q8_0.gguf?download=true) | 0.34 | 1626.49 | 0.2776 |
| [mxfp4_moe-QKO:Q6K-EUD:Q8_0](./../../resolve/main/granite-4.0-h-350m-unsloth-mxfp4_moe-QKO:Q6K-EUD:Q8_0.gguf?download=true) | 0.34 | 1642.13 | 0.3407 |

### Table - PPL Columns 

| model_name | gen | gen_er | code | code_er | math | math_er |
| ---------- | --- | ------ | ---- | ------- | ---- | ------- |
| [mxfp4_moe-EKUD:B16-O:Q6K-Q:Q8_0](./../../resolve/main/granite-4.0-h-350m-unsloth-mxfp4_moe-EKUD:B16-O:Q6K-Q:Q8_0.gguf?download=true) | 18.1560 | 0.4667 | 1.9548 | 0.0175 | 10.2986 | 0.2319 |
| [mxfp4_moe-O:Q6K-EQKUD:Q8_0](./../../resolve/main/granite-4.0-h-350m-unsloth-mxfp4_moe-O:Q6K-EQKUD:Q8_0.gguf?download=true) | 18.2304 | 0.4691 | 1.9555 | 0.0175 | 10.3074 | 0.2320 |
| [mxfp4_moe-O:Q5K-Q:Q6K-EKUD:Q8_0](./../../resolve/main/granite-4.0-h-350m-unsloth-mxfp4_moe-O:Q5K-Q:Q6K-EKUD:Q8_0.gguf?download=true) | 18.2468 | 0.4695 | 1.9583 | 0.0176 | 10.2902 | 0.2312 |
| [mxfp4_moe-QKO:Q6K-EUD:Q8_0](./../../resolve/main/granite-4.0-h-350m-unsloth-mxfp4_moe-QKO:Q6K-EUD:Q8_0.gguf?download=true) | 18.2573 | 0.4696 | 1.9580 | 0.0175 | 10.3053 | 0.2318 |
| [Q5_K](./../../resolve/main/granite-4.0-h-350m-unsloth-Q5_K.gguf?download=true) | 18.9974 | 0.4899 | 1.9842 | 0.0180 | 10.5335 | 0.2365 |

### Table - Precision Loss Columns 

| model_name | loss_general | loss_code | loss_math |
| ---------- | ------------ | --------- | --------- |
| [mxfp4_moe-EKUD:B16-O:Q6K-Q:Q8_0](./../../resolve/main/granite-4.0-h-350m-unsloth-mxfp4_moe-EKUD:B16-O:Q6K-Q:Q8_0.gguf?download=true) | 0.1368 | 0.0051 | 0.1030 |
| [mxfp4_moe-O:Q6K-EQKUD:Q8_0](./../../resolve/main/granite-4.0-h-350m-unsloth-mxfp4_moe-O:Q6K-EQKUD:Q8_0.gguf?download=true) | 0.5471 | 0.0307 | 0.1886 |
| [mxfp4_moe-O:Q5K-Q:Q6K-EKUD:Q8_0](./../../resolve/main/granite-4.0-h-350m-unsloth-mxfp4_moe-O:Q5K-Q:Q6K-EKUD:Q8_0.gguf?download=true) | 0.6376 | 0.1739 | 0.0214 |
| [mxfp4_moe-QKO:Q6K-EUD:Q8_0](./../../resolve/main/granite-4.0-h-350m-unsloth-mxfp4_moe-QKO:Q6K-EUD:Q8_0.gguf?download=true) | 0.6955 | 0.1586 | 0.1682 |
| [Q5_K](./../../resolve/main/granite-4.0-h-350m-unsloth-Q5_K.gguf?download=true) | 4.7774 | 1.4988 | 2.3863 |

---

# 🔮 **MagicQuant Hybrid Naming Scheme (Short & Practical Guide)**

MagicQuant produces hybrid quantization models where different parts of the network receive different quant schemes (e.g., MXFP4 base with BF16 embeddings, IQ4_NL attention layers, etc.).

A full tensor-by-tensor representation would be unreadable as a filename, so MagicQuant uses a **compact, modular naming scheme**:

---

## 🎯 **1. Base Format**

Every name begins with the model and its **base quantization**:

```
<ModelName>-<BaseQuant>
```

Examples:

```
Qwen3-4B-MXFP4
Qwen3-7B-IQ4_NL
```

This tells users the *default* quantization for the majority of tensors.

---

## 🎨 **2. Only Deviations Are Listed**

If certain tensor groups use a different quant scheme than the base, they appear afterwards as:

```
<GroupLetters>:<Quant>
```

Multiple group blocks can be chained:

```
<Model>-<Base>-<Groups>:<Quant>-<Groups>:<Quant>...
```

Only *exceptions* appear.
Anything not listed = uses the base quant.

---

## ✨ **3. Group Abbreviations**

These are the compact codes for each major tensor group:

| Code  | Tensor Group        | Meaning                        |
| ----- | ------------------- | ------------------------------ |
| **E** | Embeddings          | token_embd.weight              |
| **H** | LM Head             | output.weight / lm_head.weight |
| **Q** | Attention Query     | attn_q.weight                  |
| **K** | Attention Key/Value | attn_k.weight + attn_v.weight  |
| **O** | Attention Output    | attn_output.weight             |
| **U** | FFN Up/Gate         | ffn_up + ffn_gate              |
| **D** | FFN Down            | ffn_down.weight                |
| **X** | MoE Experts         | ffn_*_expert.*                 |
| **R** | MoE Router          | router.*, gate.*               |

> **You suggested “K” instead of “KV,” and yes — the shorter the better.**
> “K” represents the entire Key+Value branch (people will get it).

---

## 🧩 **4. Compressing Groups With the Same Quantization**

If multiple groups share the same quant scheme, combine them:

```
EH:B16
```

Means:

* Embeddings = BF16
* Head = BF16

Another block:

```
QKO:IQ4NL
```

Means:

* Q, K, O → IQ4_NL

You can stack as many blocks as needed.

---

## 🧪 **5. Complete Example**

### **Original Long-Form Description**

* Base: MXFP4
* Embeddings → BF16
* LM Head → BF16
* Q, K, O → IQ4_NL
* Everything else → MXFP4

### **Hybrid Name:**

```
Qwen3-4B-MXFP4-EH:B16-QKO:IQ4NL.gguf
```

This reads as:

* “Model starts as MXFP4”
* “Embeddings + Head upgraded to BF16”
* “Attention Q, K, O moved to IQ4_NL”
* “Everything else = MXFP4”

Clean. Simple. Understandable. Portable.

---

## 🔥 **6. If Only One Group Changes?**

Example: Only embeddings become Q5_K.

```
Qwen3-4B-MXFP4-E:Q5K.gguf
```

If only MoE router changes:

```
Qwen3-4B-MXFP4-R:Q8.gguf
```

---

## 🌱 **7. If EVERYTHING is the base (pure baseline)**

Just:

```
Qwen3-4B-Q4_K_M.gguf
```

No extra suffixes.

---

## 🧠 **8. Notation Guidelines (For Clarity and Aesthetics)**

* Use hyphens `-` between blocks.
* Use colon `:` between group and quant scheme.
* No need for `_` unless the quant type requires it (`IQ4_NL`, `Q4_K_M`, etc.).
* Order of groups doesn’t matter, but a consistent order is recommended:

**E, H, Q, K, O, U, D, X, R**

This mirrors information flow: embeddings → attention → ffn → moe.

---

# 🌟 **Final Example Set (For Quick Visual)**

| Description                                | Final Name                     |
| ------------------------------------------ | ------------------------------ |
| Base MXFP4, only embeddings = BF16         | `MXFP4-E:B16`                  |
| Base IQ4_NL, Q/K/O = Q6_K                  | `IQ4NL-QKO:Q6K`                |
| Base Q6_K, Head & Router = BF16            | `Q6K-HR:B16`                   |
| Everything BF16 → no hybrid                | `B16` (or `F16/BF16` baseline) |
| Full MoE override: experts + router = Q8_0 | `X R:Q8_0` → `XR:Q8_0`         |

---

### Baseline Models (Reference)

### Table - File Size + TPS + Avg Precision Loss 

| model_name | file_size_gb | bench_tps | avg_prec_loss |
| ---------- | ------------ | --------- | ------------- |
| BF16 | 0.64 | 1718.28 | 0.0000 |
| Q8_0 | 0.34 | 1598.28 | 0.3116 |
| Q6_K | 0.26 | 1513.71 | 0.5598 |
| Q5_K | 0.24 | 1305.37 | 2.8875 |
| Q4_K_M | 0.21 | 1401.44 | 12.2733 |
| IQ4_NL | 0.20 | 1679.00 | 14.2608 |
| MXFP4_MOE | 0.17 | 1713.00 | 8222.4218 |

### Table - PPL Columns 

| model_name | gen | gen_er | code | code_er | math | math_er |
| ---------- | --- | ------ | ---- | ------- | ---- | ------- |
| BF16 | 18.1312 | 0.4655 | 1.9549 | 0.0175 | 10.2880 | 0.2315 |
| Q8_0 | 18.2363 | 0.4693 | 1.9558 | 0.0175 | 10.3198 | 0.2325 |
| Q6_K | 18.3753 | 0.4719 | 1.9612 | 0.0175 | 10.2869 | 0.2294 |
| Q5_K | 18.9974 | 0.4899 | 1.9842 | 0.0180 | 10.5335 | 0.2365 |
| Q4_K_M | 21.5138 | 0.5690 | 2.0633 | 0.0194 | 11.5862 | 0.2686 |
| IQ4_NL | 22.4687 | 0.6035 | 2.0709 | 0.0194 | 11.6178 | 0.2686 |
| MXFP4_MOE | 1172.2706 | 45.9470 | 303.0942 | 7.7666 | 308.3771 | 10.9069 |

### Table - Precision Loss Columns 

| model_name | loss_general | loss_code | loss_math |
| ---------- | ------------ | --------- | --------- |
| BF16 | 0.0000 | 0.0000 | 0.0000 |
| Q8_0 | 0.5797 | 0.0460 | 0.3091 |
| Q6_K | 1.3463 | 0.3223 | 0.0107 |
| Q5_K | 4.7774 | 1.4988 | 2.3863 |
| Q4_K_M | 18.6562 | 5.5450 | 12.6186 |
| IQ4_NL | 23.9229 | 5.9338 | 12.9257 |
| MXFP4_MOE | 6365.4882 | 15404.3327 | 2897.4446 |