--- license: apache-2.0 tags: - mxfp4_hybrid - gguf - text-generation - quantized - cpu - gpu - mxfp4 - mxfp4_moe - qwen3 - magicquant - magic_quant base_model: - unsloth/granite-4.0-350m --- # MagicQuant GGUF Hybrids - granite 4.0 h 350m unsloth **(DEPRECIATED - Part of MagicQuant v1.0 which had significant flaws. Please utilize v2.0 which is production ready)** > **MagicQuant is an automated quantization, benchmarking, and evolutionary hybrid-GGUF search system for LLMs.** Each release includes models optimized to outperform standard baseline quants (Q8, Q6, Q5, Q4). If a baseline GGUF exists in this repo, the evolutionary engine couldn’t beat it. If a baseline is missing, it’s because a hybrid configuration outperformed it so completely that including the baseline would've been pointless. These hybrid GGUFs are built to be as small, fast, and low-drift as possible while preserving model capability. To dive deeper into how MagicQuant works, see the main repo: [MagicQuant on GitHub (by MagicCodingMan)](https://github.com/magiccodingman/MagicQuant-Wiki) **Notes:** * The HuggingFace hardware compatibility where it shows the bits is usually wrong. It doesn't understand hybrid mixes, so don't trust it. * Naming scheme can be found on the MagicQuant Wiki. * (tips) Less precision loss means less brain damage. More TPS means faster! Smaller is always better right? **Precision Loss Guide** * **0–0.1%** → God-tier, scientifically exact * **0.1–1%** → True near-lossless, agent-ready * **1–3%** → Minimal loss, great for personal use * **3–5%** → Borderline, but still functional * **5%+** → Toys, not tools, outside MagicQuant’s scope [Learn more about precision loss here](https://github.com/magiccodingman/MagicQuant-Wiki/blob/main/docs/precision-loss-guide.md). > **IMPORTANT NOTE:** Due to this model being so small. The test was significantly stricter in what precision loss was allowed. ### Table - File Size + TPS + Avg Precision Loss | model_name | file_size_gb | bench_tps | avg_prec_loss | | ---------- | ------------ | --------- | ------------- | | [mxfp4_moe-EKUD-B16-O-Q6K-Q-Q8_0](./../../resolve/main/granite-4.0-h-350m-unsloth-mxfp4_moe-EKUD-B16-O-Q6K-Q-Q8_0.gguf?download=true) | 0.54 | 1705.35 | 0.0816% | | [mxfp4_moe-O-Q6K-EQKUD-Q8_0](./../../resolve/main/granite-4.0-h-350m-unsloth-mxfp4_moe-O-Q6K-EQKUD-Q8_0.gguf?download=true) | 0.34 | 1605.97 | 0.2555% | ### Table - PPL Columns | model_name | gen | gen_er | code | code_er | math | math_er | | ---------- | --- | ------ | ---- | ------- | ---- | ------- | | [mxfp4_moe-EKUD-B16-O-Q6K-Q-Q8_0](./../../resolve/main/granite-4.0-h-350m-unsloth-mxfp4_moe-EKUD-B16-O-Q6K-Q-Q8_0.gguf?download=true) | 18.1560 | 0.4667 | 1.9548 | 0.0175 | 10.2986 | 0.2319 | | [mxfp4_moe-O-Q6K-EQKUD-Q8_0](./../../resolve/main/granite-4.0-h-350m-unsloth-mxfp4_moe-O-Q6K-EQKUD-Q8_0.gguf?download=true) | 18.2304 | 0.4691 | 1.9555 | 0.0175 | 10.3074 | 0.2320 * gen = ppl_general, code = ppl_code, math = ppl_math ### Table - Precision Loss Columns | model_name | loss_general | loss_code | loss_math | | ---------- | ------------ | --------- | --------- | | [mxfp4_moe-EKUD-B16-O-Q6K-Q-Q8_0](./../../resolve/main/granite-4.0-h-350m-unsloth-mxfp4_moe-EKUD-B16-O-Q6K-Q-Q8_0.gguf?download=true) | 0.1368 | 0.0051 | 0.1030 | | [mxfp4_moe-O-Q6K-EQKUD-Q8_0](./../../resolve/main/granite-4.0-h-350m-unsloth-mxfp4_moe-O-Q6K-EQKUD-Q8_0.gguf?download=true) | 0.5471 | 0.0307 | 0.1886 | * loss_* values are absolute precision-loss % vs BF16 per domain. --- ### Baseline Models (Reference) ### Table - File Size + TPS + Avg Precision Loss | model_name | file_size_gb | bench_tps | avg_prec_loss | | ---------- | ------------ | --------- | ------------- | | BF16 | 0.64 | 1718.28 | 0.0000% | | Q8_0 | 0.34 | 1598.28 | 0.3116% | | Q6_K | 0.26 | 1513.71 | 0.5598% | | Q5_K | 0.24 | 1305.37 | 2.8875% | | Q4_K_M | 0.21 | 1401.44 | 12.2733% | | IQ4_NL | 0.20 | 1679.00 | 14.2608% | | MXFP4_MOE | 0.17 | 1713.00 | 8222.4218% | ### Table - PPL Columns | model_name | gen | gen_er | code | code_er | math | math_er | | ---------- | --- | ------ | ---- | ------- | ---- | ------- | | BF16 | 18.1312 | 0.4655 | 1.9549 | 0.0175 | 10.2880 | 0.2315 | | Q8_0 | 18.2363 | 0.4693 | 1.9558 | 0.0175 | 10.3198 | 0.2325 | | Q6_K | 18.3753 | 0.4719 | 1.9612 | 0.0175 | 10.2869 | 0.2294 | | Q5_K | 18.9974 | 0.4899 | 1.9842 | 0.0180 | 10.5335 | 0.2365 | | Q4_K_M | 21.5138 | 0.5690 | 2.0633 | 0.0194 | 11.5862 | 0.2686 | | IQ4_NL | 22.4687 | 0.6035 | 2.0709 | 0.0194 | 11.6178 | 0.2686 | | MXFP4_MOE | 1172.2706 | 45.9470 | 303.0942 | 7.7666 | 308.3771 | 10.9069 | * gen = ppl_general, code = ppl_code, math = ppl_math ### Table - Precision Loss Columns | model_name | loss_general | loss_code | loss_math | | ---------- | ------------ | --------- | --------- | | BF16 | 0.0000 | 0.0000 | 0.0000 | | Q8_0 | 0.5797 | 0.0460 | 0.3091 | | Q6_K | 1.3463 | 0.3223 | 0.0107 | | Q5_K | 4.7774 | 1.4988 | 2.3863 | | Q4_K_M | 18.6562 | 5.5450 | 12.6186 | | IQ4_NL | 23.9229 | 5.9338 | 12.9257 | | MXFP4_MOE | 6365.4882 | 15404.3327 | 2897.4446 | * loss_* values are absolute precision-loss % vs BF16 per domain. ## Support I’m a solo developer working full time for myself to achieve my dream, pouring nights and weekends into open protocols and tools that I hope make the world a little better. If you chip in, you're helping me keep the lights on while I keep shipping. [Click here to see ways to support](https://sayou.biz/support) - BTC, Paypal, GitHub sponsors. Or, just drop a like on the repo :)