Swift-Qwen3.8-27B-Uncensored-GPTQ-Int4-sym-G128-MTP-BF16

GPTQ-Int4 quantization of d0xin/Swift-Qwen3.8-27B-Uncensored-BF16, built for Intel Arc / vLLM XPU with the MTP head preserved in BF16 so native speculative decoding still works.

The recipe is a reproduction of kernelogic/Qwen3.8-27B-Uncensored-GPTQ-Int4-sym-G128-MTP-BF16, applied to the Swift lineage instead of the JonathanColetti one. Same quantizer version, same config, same MTP handling.

Why this exists

Quantizing this model naively breaks speculative decoding. The 15 mtp.* tensors are the draft head; if they get quantized along with everything else, draft acceptance collapses and you lose roughly half your decode speed. The fix is one line of quantize config:

dynamic={"-:.*mtp.*": {}}   # exclude mtp.* from quantization -> stays BF16

The result has 400 quantized weight tensors (I32) + 15 preserved BF16 MTP tensors.

There is a second, less obvious requirement. transformers' Qwen3_5 declares _keys_to_ignore_on_load_unexpected = [r"^mtp.*"], so the draft tensors are dropped at load and never become modules — the dynamic exclusion on its own emits a checkpoint with no draft head at all, silently. What writes them back is gptqmodel's out_of_model_tensors = {"prefixes": ["mtp"]}, which re-reads them from the source safetensors at save time. That is also why they stay BF16 while the body is fp16: they are copied bytes, never converted.

Provenance

Qwen/Qwen3.8-27B                          (base)
  └─ ukisai/Swift-Qwen3.8-27b                 (Swift, efficient-thinking finetune)
       └─ d0xin/Swift-Qwen3.8-27B-Uncensored-BF16
            (uncensored, rank-1 directional ablation, BF16)
              └─ this repo                             (GPTQ-Int4, 20 GB)

Quantization

Tool	gptqmodel 7.3.2
Config	bits=4, group_size=128, sym=True, desc_act=False, lm_head=False, pack_dtype=int32
MTP	dynamic={"-:.*mtp.*": {}} → 15 tensors kept BF16
Calibration	256 samples @ 2048 tokens from allenai/c4 (general web text)
Time	22.2 min of per-module quantization across 64 layers (sum of quant_log.csv; excludes calibration forwards, offload and packing, so wall clock is higher)
Output	5 shards, 20 GB, 2399 tensors

Group size is 128, verified from the tensor shapes (qweight.rows * 8 / scales.rows) rather than taken from the config's own declaration. lm_head and the vision tower (333 tensors) are left unquantized.

Measured performance

llama-benchy --pp 4096 --tg 256 --depth 0 across concurrency levels 1 2 4 8, prefill 2350.21 tok/s at c1. 48 SpecDecoding samples over the run.

Workload	tok/s	MTP acceptance
Prose, single stream (c1)	79.34 total	3.14 (range 2.00-6.00)
Prose, 2 concurrent (c2)	91.77 total, 53.39/req, 1.16x scaling
Prose, 4 concurrent (c4)	159.70 total, 47.13/req, 2.01x scaling
Prose, 8 concurrent (c8)	117.21 total, 27.04/req, 1.48x scaling

Prefill is flat across concurrency and decode is not. One stream already saturates prefill; decode scales sublinearly and turns over once prefill/decode interference dominates. Per-request decode is the number a single user experiences.

Per-position acceptance: 0.716, 0.504, 0.367, 0.307, 0.248. Judge any MTP depth change on this curve, never on tok/s -- a decay ratio near 0.8 pays for the extra drafter pass and near 0.68 is break-even.

Serving (vLLM XPU)

vllm serve /model \
  --quantization gptq --dtype float16 \
  --kv-cache-dtype fp8 \
  --max-num-batched-tokens 8192 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":5}' \
  --chat-template /model/chat_template.jinja

Qwen3.8 MTP on XPU needs a patched vLLM. The build used to produce and serve this checkpoint is at github.com/greglechin/vllm-xpu; the Intel Arc Pro B70 cookbook is the other published route. Context length, --tensor-parallel-size and --gpu-memory-utilization are deliberately omitted above because they depend on your card count and VRAM — size them from your own boot log.

Three things that will bite you

1. MTP depth is not free, and tok/s is the wrong way to judge it. Read the per-position acceptance curve from the engine's SpecDecoding metrics, not the run mean: a decay ratio around 0.8 pays for the extra drafter pass and around 0.68 is break-even. Changing depth typically moves throughput by less than boot-to-boot variance, so a single A/B run measures nothing.
2. Prefix caching depends on your engine build. Qwen3.8 is a hybrid GDN/Mamba architecture and upstream vLLM does not declare supports_mamba_prefix_caching for it, which is why the reference checkpoint advises disabling it (measured there: 3,036 queries → 0 hits). With the mamba-align patches and --mamba-cache-mode align it does work — 84.8–85.3% hit rate measured on the serving stack above. Check your own hit-rate metric rather than assuming either outcome.
3. --kv-cache-dtype fp8 is effectively required on 32 GB cards at long context; fp16 KV does not fit.

Limitations — please read

• No quality evaluations were run on this quantization. No perplexity comparison against the BF16 source, no coding or reasoning benchmarks, no quantitative refusal-rate testing. What was verified is structural: tensor counts and dtypes, the preserved BF16 MTP head, complete quantized-module tensor sets, group size derived from shapes, and an unquantized lm_head and vision tower. If you need quality guarantees, measure before relying on it.
• Calibration used general web text with no code. If coding quality matters to you, a code-inclusive calibration set would likely be better. This is inherited from the reference recipe, which names the same limitation.
• Uncensoring is inherited, not verified here. All refusal-removal properties come from the upstream ablation (d0xin/Swift-Qwen3.8-27B-Uncensored-BF16 reports 0/100 refusals on its own fixed evaluation); this repo only changes numeric precision. Quantization was not evaluated for its effect on that behaviour in either direction.
• The upstream's capability evidence is thin, and it stacks with the point above. Its INTELLIGENCE_VALIDATION.json reports 298 paired samples with a 95% CI of [−1.68, +5.03] pp — so a ~4% relative capability loss sits inside the interval and is not excluded. McNemar there operates on only 27 discordant pairs, its MATH-500 arm scores 8% on both models (a broken harness cannot detect damage, and it supplies a third of the samples), and the agentic, tool-calling, multimodal and long-context capabilities the upstream card claims to preserve are not evaluated at all. Measure on your own workload before trusting either layer.
• This is an uncensored model. It will attempt requests an aligned model declines. You are responsible for how you use it.

Licence

Not Apache-2.0. These weights inherit the Swift Open License v1.0 from ukisai/Swift-Qwen3.8-27b. Personal, research, educational, evaluation and commercial use are free for individuals and organizations with annual recurring revenue, including affiliates, of up to US$1,000,000. Above that threshold, commercial use requires a separate Swift Enterprise License — contact UkisAI.

Credits

• Qwen — base model
• UkisAI — Swift finetune
• d0xin — directional ablation
• kernelogic/Qwen3.8-27B-Uncensored-GPTQ-Int4-sym-G128-MTP-BF16 — the quantization recipe this reproduces
• gptqmodel — quantization