Qwen3.6-35B-A3B-BUDGET-GGUF

⚠️ Experimental. Use with caution. Both files in this repo are research artifacts from a size-target quantization + QAT experiment. They have been evaluated on perplexity only (all measurements below). No instruction-following benchmarks, no agentic evals, no human testing. They are aggressive low-bit builds and will behave worse than the standard tiers. Verify outputs before relying on them.

Derivative of Qwen/Qwen3.6-35B-A3B, compressed to a hard 13.5 GiB byte budget so that the full model plus 64k context fits in 16 GiB of GPU memory (measured on unified memory; see the fit section for what that does and does not prove). If you have more memory, use the standard tiers instead: MagicQuant hybrids (Q4/Q5/Q6) or the ROCmFPX build.

The experiment

The smallest standard tier of this model is 20.2 GiB (21.7 GB). It cannot fit in 16 GiB at any context length, so this repo asks a different question: what is the best 35B-A3B you can have if the budget is fixed at 16 GiB total, including 64k of context? (Sizes on this card are GiB, 1024-based, throughout.)

The context arithmetic is what makes it plausible at all. Qwen3.6-35B-A3B is a hybrid-attention model: only 10 of its 40 layers are full attention (2 KV heads x 256 head dim); the other 30 are linear-attention layers whose state does not grow with context. KV cache at 64k is therefore about 1.25 GiB at f16, several times smaller than a dense model of this size. 16 GB minus KV minus runtime buffers leaves roughly 13.5 GiB for weights, a 0.20 size ratio versus BF16, well below the Q4 band.

The weights were fitted to that budget with MagicQuant v2's size-target search: an exact per-tensor knapsack (753 tensor assignments, imatrix-calibrated) under a hard byte ceiling, verified against real perplexity rather than a proxy. The file landed at 13.51 GiB against a 13.5 GiB request.

A second copy of the model then went through quantization-aware training (QAT): frozen-mode LoRA trained against this exact quantization layout, merged, and re-packed at the identical 753-tensor allocation. Both copies are published because they win on different workloads (measurements below).

The two files

File	Size	What it is
Qwen3.6-35B-A3B-BUDGET-13.5GiB.gguf	13.51 GiB	Budget-search output, no QAT
Qwen3.6-35B-A3B-BUDGET-13.5GiB-QAT.gguf	13.51 GiB	Same per-tensor layout, weights adjusted by quantization-aware LoRA

Which one should you use? The non-QAT file is the recommended default. The QAT file is published as a research artifact: it measured 2.20% better perplexity on held-out chat-formatted data (the distribution it was trained on), while the non-QAT file measured 0.47% better on raw wikitext. Be aware of how weak that evidence is: both deltas sit within about 1.3 standard errors of the reported measurement uncertainty, so the domain split is suggestive, not statistically established. If your workload is chat/agentic and you are willing to run an experimental variant, try the QAT file and judge on your own tasks.

Does it actually fit 16 GiB at 64k? (measured, with scope)

Measured by GPU-memory (GTT) delta on an AMD Strix Halo APU (unified memory), model fully offloaded, 64k context allocated:

KV cache	Total footprint	Verdict
f16	16.06 GiB	just over
q8_0 (-fa on -ctk q8_0 -ctv q8_0)	15.47 GiB	fits

The 16 GiB claim requires the quantized KV cache flags. Two scope caveats:

• This was measured on unified memory, not on a discrete 16 GB card. A real dGPU reserves some VRAM for driver and display; 15.47 GiB leaves only ~0.5 GiB of headroom, so on a discrete 16 GB card the fit is thin and untested. Headless cards and iGPU-display systems have the best odds.
• Generation speed on the same hardware: 26.9 tok/s with q8 KV (23.4 with f16 KV), prompt processing 79-91 tok/s. These are short-prompt, near-empty-context speeds; the 64k context was allocated, not filled. Throughput at deep context was not measured and will be lower.

Measured quality

All perplexity with llama-perplexity, -c 512. Lower is better.

Out of domain, wikitext-2 (raw encyclopedic prose):

Build	PPL (Strix Halo, full corpus)	PPL (RTX 3090, 100 chunks)
BF16 baseline	6.7208	n/a
BUDGET-13.5GiB	7.2809 (+8.33% vs BF16)	7.1882 ± 0.112
BUDGET-13.5GiB-QAT	7.3150 (+8.84% vs BF16)	7.2333 ± 0.111

In domain, held-out chat blend (297 conversations rendered through the model's own chat template; a row-disjoint resample from the same synthetic generator and bucket weights as the QAT training data, verified 0.00% exact and normalized row overlap; 100 chunks):

Build	PPL
BUDGET-13.5GiB	2.7127 ± 0.033
BUDGET-13.5GiB-QAT	2.6529 ± 0.031 (−2.20%)

Three honest readings of these tables:

1. +8-9% wikitext PPL versus BF16 is real degradation. This is an aggressive build. A budget name is a size claim, not a quality tier; the standard Q4/Q5/Q6 hybrids are measurably better models if you have the memory for them.
2. The per-domain differences between the two files are small relative to their error bars (within ~1.3 standard errors in-domain, well within one standard error on the 3090 wikitext run). Treat the domain split as a directional observation, not an established result. No bf16-control arm was run at 35B, so domain adaptation and quantization compensation are not separated at this scale.
3. The direction is consistent with a controlled experiment at small scale. A 4-arm study (writeup) showed frozen-mode QAT recovers real quantization damage against a bf16+identical-LoRA control (+13.0%, vs +21.8% for live mode), while the identical LoRA by itself raised raw-prose perplexity, which is why a frozen run's raw wikitext delta can go negative even when its controlled recovery is positive. The wikitext regression above matches that signature; the in-domain improvement here is a new, uncontrolled observation from this repo's own two measurements.

QAT details

• Mode: frozen (the expert base is fake-quantized once at wrap time; a LoRA delta trains on top). Live mode re-quantizes every forward pass and measures about 1.7x the recovery rate (+21.8% vs +13.0% confound-controlled), but is computationally infeasible at this scale (about 92 minutes per forward pass on the fused expert tensors).
• Adapters: r=32 / alpha=64 on attention and dense projections; r=4 / alpha=8 on the 80 fused 3-D MoE expert tensors; 281M trainable parameters total.
• Training: 500 steps, lr 2e-4 cosine, sequence length 512, on a 6,000-conversation blend (30% code, 30% math with chain-of-thought, 25% agentic tool-calling, 10% general chat, 5% multilingual).
• Packing: adapters merged into the base weights, then re-packed at the identical per-tensor allocation the budget search solved for. The shipped GGUF is exact-ggml (packed with libggml, byte-identical to llama-quantize output for each tensor type).

Quantization method

This is not a uniform quantization, and it is not one of MagicQuant's standard tiers. The v2 size-target search assigns a scheme to every tensor individually to minimize predicted quality loss under the byte ceiling. What actually ships, by tensor count: 228 F32, 142 Q8_0, 116 Q6_K, 78 Q2_K, 76 Q3_K, 67 Q5_K, 40 F16, 6 Q4_K (753 total). The bulk of the size lives in the MoE expert tensors at Q2_K/Q3_K, which is where most of the quality cost comes from. Norms stay at F32, the router at Q8_0/Q6_K, and the SSM state operands (ssm_a / ssm_dt / conv) at F32/Q8_0; the SSM output projections, by contrast, are among the lowest-precision non-expert tensors in the file (Q2_K/Q4_K), a knapsack outcome, not an oversight.

Search and packing by MagicQuant. Built from the BF16 release weights (not re-quantized from another GGUF).

MTP / speculative decoding note

The MTP draft tensors (blk.40.nextn.*) are present in both files. MTP serving was not measured on this build; it allocates an additional draft context on top of the figures above, and this build's whole premise is ~0.5 GiB of headroom, so budget for that before enabling it. If MTP speculative serving is your goal and you have the memory, the standard tiers document a measured 1.6-1.9x speedup.

Usage

llama.cpp (the KV flags are required for the 16 GiB claim; swap in the QAT filename if that is the variant you chose):

llama-server -m Qwen3.6-35B-A3B-BUDGET-13.5GiB.gguf \
  -c 65536 -fa on -ctk q8_0 -ctv q8_0 -ngl 999 --jinja

LM Studio: untested. These measurements all used a recent llama.cpp build; the qwen3.5/3.6 MoE hybrid-attention architecture needs a runtime new enough to support it. If it loads for you, setting KV cache quantization to Q8_0 is the equivalent of the flags above (advice, not a measurement).

Caveats and limitations

• Experimental and untested beyond perplexity. No benchmark suite, no agentic eval, no human eval has been run on either file. Use with caution and verify outputs.
• Aggressive quantization. Expect weaker long-tail factual recall and more brittle code generation than the standard tiers; the +8-9% wikitext gap versus BF16 is a floor on the damage, not a ceiling.
• Text-only. The base model is multimodal, but these GGUFs ship without a vision projector (no mmproj file); image input is not available.
• QAT is lightly trained. 500 steps on a synthetic-leaning blend; it may bias style toward that blend, and its advantage is measured on perplexity, not on task success.
• The base model's Apache-2.0 license and its biases and limitations apply.

---

Generated with MagicQuant. Frozen-mode QAT methodology and the controlled experiment behind the domain-split guidance: docs/experiments/qat-frozen-mode-2026-08.md.