---
license: apache-2.0
library_name: llama.cpp
base_model:
- Qwen/Qwen3.6-35B-A3B
base_model_relation: quantized
pipeline_tag: text-generation
quantized_by: MagicQuant
tags:
- gguf
- quantized
- magicquant
- qat
- experimental
- budget-build
---

# Qwen3.6-35B-A3B-BUDGET-GGUF

> ⚠️ **Experimental. Use with caution.** Both files in this repo are research artifacts from a size-target quantization + QAT experiment. They have been evaluated on **perplexity only** (all measurements below). No instruction-following benchmarks, no agentic evals, no human testing. They are aggressive low-bit builds and will behave worse than the standard tiers. Verify outputs before relying on them.

Derivative of [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B), compressed to a hard 13.5 GiB byte budget so that **the full model plus 64k context fits in 16 GiB of GPU memory** (measured on unified memory; see the fit section for what that does and does not prove). If you have more memory, use the standard tiers instead: [MagicQuant hybrids (Q4/Q5/Q6)](https://huggingface.co/lmcoleman/Qwen3.6-35B-A3B-MagicQuant-GGUF) or the [ROCmFPX build](https://huggingface.co/lmcoleman/Qwen3.6-35B-A3B-ROCmFPX-GGUF).

## The experiment

The smallest standard tier of this model is 20.2 GiB (21.7 GB). It cannot fit in 16 GiB at any context length, so this repo asks a different question: what is the best 35B-A3B you can have if the budget is fixed at 16 GiB *total*, including 64k of context? (Sizes on this card are GiB, 1024-based, throughout.)

The context arithmetic is what makes it plausible at all. Qwen3.6-35B-A3B is a hybrid-attention model: only 10 of its 40 layers are full attention (2 KV heads x 256 head dim); the other 30 are linear-attention layers whose state does not grow with context. KV cache at 64k is therefore about 1.25 GiB at f16, several times smaller than a dense model of this size. 16 GB minus KV minus runtime buffers leaves roughly 13.5 GiB for weights, a 0.20 size ratio versus BF16, well below the Q4 band.

The weights were fitted to that budget with MagicQuant v2's size-target search: an exact per-tensor knapsack (753 tensor assignments, imatrix-calibrated) under a hard byte ceiling, verified against real perplexity rather than a proxy. The file landed at 13.51 GiB against a 13.5 GiB request.

A second copy of the model then went through **quantization-aware training** (QAT): frozen-mode LoRA trained against this exact quantization layout, merged, and re-packed at the identical 753-tensor allocation. Both copies are published because they win on different workloads (measurements below).

## The two files

| File | Size | What it is |
|---|---|---|
| [Qwen3.6-35B-A3B-BUDGET-13.5GiB.gguf](https://huggingface.co/lmcoleman/Qwen3.6-35B-A3B-BUDGET-GGUF/blob/main/Qwen3.6-35B-A3B-BUDGET-13.5GiB.gguf) | 13.51 GiB | Budget-search output, no QAT |
| [Qwen3.6-35B-A3B-BUDGET-13.5GiB-QAT.gguf](https://huggingface.co/lmcoleman/Qwen3.6-35B-A3B-BUDGET-GGUF/blob/main/Qwen3.6-35B-A3B-BUDGET-13.5GiB-QAT.gguf) | 13.51 GiB | Same per-tensor layout, weights adjusted by quantization-aware LoRA |

**Which one should you use?** The **non-QAT file is the recommended default**. The QAT file is published as a research artifact: it measured 2.20% better perplexity on held-out chat-formatted data (the distribution it was trained on), while the non-QAT file measured 0.47% better on raw wikitext. Be aware of how weak that evidence is: both deltas sit within about 1.3 standard errors of the reported measurement uncertainty, so the domain split is **suggestive, not statistically established**. If your workload is chat/agentic and you are willing to run an experimental variant, try the QAT file and judge on your own tasks.

## Does it actually fit 16 GiB at 64k? (measured, with scope)

Measured by GPU-memory (GTT) delta on an AMD Strix Halo APU (unified memory), model fully offloaded, 64k context allocated:

| KV cache | Total footprint | Verdict |
|---|---|---|
| f16 | 16.06 GiB | just over |
| q8_0 (`-fa on -ctk q8_0 -ctv q8_0`) | **15.47 GiB** | **fits** |

The 16 GiB claim **requires the quantized KV cache flags**. Two scope caveats:

- This was measured on **unified memory**, not on a discrete 16 GB card. A real dGPU reserves some VRAM for driver and display; 15.47 GiB leaves only ~0.5 GiB of headroom, so on a discrete 16 GB card the fit is **thin and untested**. Headless cards and iGPU-display systems have the best odds.
- Generation speed on the same hardware: 26.9 tok/s with q8 KV (23.4 with f16 KV), prompt processing 79-91 tok/s. These are **short-prompt, near-empty-context speeds**; the 64k context was allocated, not filled. Throughput at deep context was not measured and will be lower.

## Measured quality

All perplexity with llama-perplexity, `-c 512`. Lower is better.

**Out of domain, wikitext-2** (raw encyclopedic prose):

| Build | PPL (Strix Halo, full corpus) | PPL (RTX 3090, 100 chunks) |
|---|---|---|
| BF16 baseline | 6.7208 | n/a |
| BUDGET-13.5GiB | 7.2809 (+8.33% vs BF16) | 7.1882 ± 0.112 |
| BUDGET-13.5GiB-QAT | 7.3150 (+8.84% vs BF16) | 7.2333 ± 0.111 |

**In domain, held-out chat blend** (297 conversations rendered through the model's own chat template; a row-disjoint resample from the *same synthetic generator and bucket weights* as the QAT training data, verified 0.00% exact and normalized row overlap; 100 chunks):

| Build | PPL |
|---|---|
| BUDGET-13.5GiB | 2.7127 ± 0.033 |
| BUDGET-13.5GiB-QAT | **2.6529 ± 0.031** (−2.20%) |

Three honest readings of these tables:

1. **+8-9% wikitext PPL versus BF16 is real degradation.** This is an aggressive build. A budget name is a size claim, not a quality tier; the standard Q4/Q5/Q6 hybrids are measurably better models if you have the memory for them.
2. **The per-domain differences between the two files are small relative to their error bars** (within ~1.3 standard errors in-domain, well within one standard error on the 3090 wikitext run). Treat the domain split as a directional observation, not an established result. No bf16-control arm was run at 35B, so domain adaptation and quantization compensation are not separated at this scale.
3. **The direction is consistent with a controlled experiment at small scale.** A 4-arm study ([writeup](https://github.com/lucasmcoleman/MagicQuant/blob/master/docs/experiments/qat-frozen-mode-2026-08.md)) showed frozen-mode QAT recovers real quantization damage against a bf16+identical-LoRA control (+13.0%, vs +21.8% for live mode), while the identical LoRA *by itself* raised raw-prose perplexity, which is why a frozen run's raw wikitext delta can go negative even when its controlled recovery is positive. The wikitext regression above matches that signature; the in-domain improvement here is a new, uncontrolled observation from this repo's own two measurements.

## QAT details

- **Mode**: frozen (the expert base is fake-quantized once at wrap time; a LoRA delta trains on top). Live mode re-quantizes every forward pass and measures about 1.7x the recovery rate (+21.8% vs +13.0% confound-controlled), but is computationally infeasible at this scale (about 92 minutes per forward pass on the fused expert tensors).
- **Adapters**: r=32 / alpha=64 on attention and dense projections; r=4 / alpha=8 on the 80 fused 3-D MoE expert tensors; 281M trainable parameters total.
- **Training**: 500 steps, lr 2e-4 cosine, sequence length 512, on a 6,000-conversation blend (30% code, 30% math with chain-of-thought, 25% agentic tool-calling, 10% general chat, 5% multilingual).
- **Packing**: adapters merged into the base weights, then re-packed at the identical per-tensor allocation the budget search solved for. The shipped GGUF is exact-ggml (packed with libggml, byte-identical to llama-quantize output for each tensor type).

## Quantization method

This is **not a uniform quantization**, and it is not one of MagicQuant's standard tiers. The v2 size-target search assigns a scheme to every tensor individually to minimize predicted quality loss under the byte ceiling. What actually ships, by tensor count: 228 F32, 142 Q8_0, 116 Q6_K, 78 Q2_K, 76 Q3_K, 67 Q5_K, 40 F16, 6 Q4_K (753 total). The bulk of the size lives in the MoE expert tensors at Q2_K/Q3_K, which is where most of the quality cost comes from. Norms stay at F32, the router at Q8_0/Q6_K, and the SSM state operands (ssm_a / ssm_dt / conv) at F32/Q8_0; the SSM *output projections*, by contrast, are among the lowest-precision non-expert tensors in the file (Q2_K/Q4_K), a knapsack outcome, not an oversight.

Search and packing by [MagicQuant](https://github.com/lucasmcoleman/MagicQuant). Built from the BF16 release weights (not re-quantized from another GGUF).

## MTP / speculative decoding note

The MTP draft tensors (`blk.40.nextn.*`) are present in both files. MTP serving was **not measured on this build**; it allocates an additional draft context on top of the figures above, and this build's whole premise is ~0.5 GiB of headroom, so budget for that before enabling it. If MTP speculative serving is your goal and you have the memory, the [standard tiers](https://huggingface.co/lmcoleman/Qwen3.6-35B-A3B-MagicQuant-GGUF) document a measured 1.6-1.9x speedup.

## Usage

llama.cpp (the KV flags are required for the 16 GiB claim; swap in the QAT filename if that is the variant you chose):

```bash
llama-server -m Qwen3.6-35B-A3B-BUDGET-13.5GiB.gguf \
  -c 65536 -fa on -ctk q8_0 -ctv q8_0 -ngl 999 --jinja
```

LM Studio: **untested.** These measurements all used a recent llama.cpp build; the qwen3.5/3.6 MoE hybrid-attention architecture needs a runtime new enough to support it. If it loads for you, setting KV cache quantization to Q8_0 is the equivalent of the flags above (advice, not a measurement).

## Caveats and limitations

- **Experimental and untested beyond perplexity.** No benchmark suite, no agentic eval, no human eval has been run on either file. Use with caution and verify outputs.
- **Aggressive quantization.** Expect weaker long-tail factual recall and more brittle code generation than the standard tiers; the +8-9% wikitext gap versus BF16 is a floor on the damage, not a ceiling.
- **Text-only.** The base model is multimodal, but these GGUFs ship without a vision projector (no mmproj file); image input is not available.
- **QAT is lightly trained.** 500 steps on a synthetic-leaning blend; it may bias style toward that blend, and its advantage is measured on perplexity, not on task success.
- The base model's Apache-2.0 license and its biases and limitations apply.

---

*Generated with [MagicQuant](https://github.com/lucasmcoleman/MagicQuant). Frozen-mode QAT methodology and the controlled experiment behind the domain-split guidance: [docs/experiments/qat-frozen-mode-2026-08.md](https://github.com/lucasmcoleman/MagicQuant/blob/master/docs/experiments/qat-frozen-mode-2026-08.md).*