Add MTP weights + speculative decoding support

- mtp.safetensors: 19 MTP tensors in per-expert split format (768 expert keys + 17 non-expert)
- model.safetensors.index.json: weight_map updated with 19 MTP keys, total_size 22.7GB
- config.json: add mlp_only_layers, modules_to_not_convert (Qwen official schema parity)
- README: document vLLM 0.19.1 + SGLang 0.5.10 MTP configs + verified perf metrics

Verified speculative decoding:
- vLLM 0.19.1 + 4× RTX 3060: 56-82 t/s, 70-89% acceptance rate
- SGLang 0.5.10 EAGLE: 60-105 t/s, 36-66% acceptance

vLLM users: do NOT pass --quantization moe_wna16 (auto-detect required).

README.md

@@ -13,6 +13,8 @@ tags:
 - moe
 - qwen3.6
 - qwen3-5-moe
+- mtp
+- speculative-decoding
 library_name: transformers
 pipeline_tag: image-text-to-text
 ---
@@ -21,6 +23,8 @@ pipeline_tag: image-text-to-text
 
 GPTQ Int4 quantization of [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B), produced on **consumer multi-GPU hardware** (4× RTX 3060 12GB) using Python 3.13t free-threading.
 
+This v2 release ships **MTP (Multi-Token Prediction) speculative decoding weights** verified working on both vLLM 0.19.1 and SGLang 0.5.10.
+
 ## Quality
 
 | Metric | Value |
@@ -31,6 +35,7 @@ GPTQ Int4 quantization of [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwe
 | Loss median | 9.29e-05 |
 | Loss max | 2.14e-03 |
 | Total modules | 30,720 |
+| Perplexity (wikitext-2-raw-v1) | 6.1846 (~97.9% BF16 retention) |
 
 ## Model specs
 
@@ -42,8 +47,9 @@ GPTQ Int4 quantization of [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwe
 | Hidden layers | 40 |
 | Context length | 262,144 tokens |
 | Quantization | GPTQ v2, 4-bit, group_size=128, symmetric |
-| Quantized size | 21.2 GB (68.4% reduction from bf16) |
+| Quantized size | 22.7 GB (incl. MTP weights) |
 | KV cache support | fp16, bf16, fp8_e4m3 (storage-only on Ampere) |
+| MTP head | Included (BF16, 785 keys, split per-expert format) |
 
 ### What's quantized vs kept bf16
 
@@ -53,7 +59,7 @@ GPTQ Int4 quantization of [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwe
 - Attention layers (`*.attn.*`)
 - MoE routers (`*.mlp.gate`)
 - Shared experts (`*.shared_expert.*`)
-- Multi-token prediction heads (`*.mtp.*`)
+- Multi-token prediction heads (`*.mtp.*`) — see [Speculative decoding](#speculative-decoding-mtp) below
 - Vision encoder (`*.visual.*`)
 - Embeddings and `lm_head`
 
@@ -111,16 +117,83 @@ Verified on 4× RTX 3060 12GB with SGLang 0.5.10 (`max_total_num_tokens=415184`,
 
 ```bash
 vllm serve palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4 \
-  --quantization gptq_marlin \
   --tensor-parallel-size 4 \
-  --max-model-len 131072 \
-  --gpu-memory-utilization 0.85
+  --max-model-len 200000 \
+  --gpu-memory-utilization 0.85 \
+  --kv-cache-dtype fp8 \
+  --dtype bfloat16 \
+  --reasoning-parser qwen3 \
+  --enable-auto-tool-choice \
+  --tool-call-parser qwen3_coder \
+  --trust-remote-code
 ```
 
+> **Important:** Do **not** pass `--quantization moe_wna16` to vLLM. Let vLLM auto-detect quantization from `config.json`. Forcing the flag triggers a `KeyError: 'experts.w2_weight'` in the `Qwen3_5MoeMTP` loader path even when MTP is disabled.
+
 ### Transformers (single-GPU, for testing)
 
 Requires `trust_remote_code=True` for the Qwen3.5-MoE architecture.
 
+## Speculative decoding (MTP)
+
+This release ships **MTP (Multi-Token Prediction) weights** in the per-expert split format expected by vLLM and SGLang loaders (785 MTP keys total, all BF16). Speculative decoding is verified working on both engines.
+
+### vLLM with MTP
+
+```bash
+vllm serve palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4 \
+  --tensor-parallel-size 4 \
+  --max-model-len 200000 \
+  --gpu-memory-utilization 0.85 \
+  --kv-cache-dtype fp8 \
+  --dtype bfloat16 \
+  --reasoning-parser qwen3 \
+  --enable-auto-tool-choice \
+  --tool-call-parser qwen3_coder \
+  --speculative-config '{"method": "mtp", "num_speculative_tokens": 2}' \
+  --trust-remote-code
+```
+
+Verified on vLLM 0.19.1 + 4× RTX 3060 12GB:
+
+| Metric | Value |
+|--------|-------|
+| Steady-state decode | 56–82 t/s |
+| Avg draft acceptance rate | 70–89% (peak 88.9%) |
+| Per-position acceptance | token 1: ~0.93, token 2: ~0.85 |
+| Mean acceptance length | 2.4–2.8 / 2 draft tokens |
+| KV pool | 160,800 tokens (262K max-model-len, 2.34× concurrency) |
+
+### SGLang with MTP (EAGLE)
+
+```bash
+SGLANG_ENABLE_SPEC_V2=1 \
+SGLANG_MAMBA_CONV_DTYPE=float16 \
+python -m sglang.launch_server \
+  --model-path palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4 \
+  --tp-size 4 \
+  --dtype float16 \
+  --quantization moe_wna16 \
+  --context-length 200000 \
+  --mem-fraction-static 0.80 \
+  --mamba-scheduler-strategy extra_buffer \
+  --speculative-algorithm EAGLE \
+  --speculative-eagle-topk 1 \
+  --speculative-num-steps 3 \
+  --speculative-num-draft-tokens 4 \
+  --reasoning-parser qwen3 \
+  --tool-call-parser qwen3_coder \
+  --port 30000
+```
+
+Verified on SGLang 0.5.10: 60–105 t/s decode, 36–66% acceptance.
+
+### Notes on MTP weight layout
+
+This release stores MTP experts in **per-expert split format** (`mtp.layers.0.mlp.experts.{i}.{gate,up,down}_proj.weight`, 256 experts × 3 projections = 768 expert keys), matching the layout that the Qwen official `Qwen3.5-35B-A3B-GPTQ-Int4` checkpoint and the upstream vLLM/SGLang MTP loaders expect.
+
+The Qwen3.6 BF16 base model stores these as **fused 3D tensors** (`gate_up_proj` shape `[E, 2I, H]`, `down_proj` shape `[E, H, I]`). They are split bit-for-bit during release packaging — there is no quantization or numerical transformation of the MTP head, only a tensor reshape.
+
 ## Reproducibility
 
 The full per-module quantization log is published as [`quant_log.csv`](./quant_log.csv) (1.8 MB). Each row records the layer, module, GPTQ loss (or RTN fallback marker), sample count, damping value, and wall-clock time — making the run fully auditable.
@@ -130,6 +203,7 @@ The full per-module quantization log is published as [`quant_log.csv`](./quant_l
 - **Per-layer RTN distribution.** Layers 32–39 show ~7–10% RTN rate vs ~1–3% for early layers. Consistent with MoE routing concentration in deeper layers and memory pressure late in the run.
 - **Cold experts.** 157 of 256 expert IDs fell back to RTN in at least one layer; 99 always got GPTQ. Top cold experts: 235, 249, 234, 197, 237. These rarely route at inference.
 - **Ampere-only limitations.** On RTX 3060 (SM86), fp8 KV cache is storage-only (dequantized for attention). No FP8 compute path.
+- **vLLM `--quantization` flag.** Do not pass `--quantization moe_wna16` to vLLM — it triggers a `KeyError` in the MTP loader path. SGLang requires the flag; vLLM must auto-detect.
 
 ## Credits
 

config.json

@@ -56,7 +56,8 @@
     "method": "gptq",
     "pack_dtype": "int32",
     "quant_method": "gptq",
-    "sym": true
+    "sym": true,
+    "modules_to_not_convert": []
   },
   "rope_parameters": {
     "rope_theta": 10000.0,
@@ -151,7 +152,8 @@
     "shared_expert_intermediate_size": 512,
     "tie_word_embeddings": false,
     "use_cache": true,
-    "vocab_size": 248320
+    "vocab_size": 248320,
+    "mlp_only_layers": []
   },
   "tie_word_embeddings": false,
   "transformers_version": "5.5.4",
@@ -176,4 +178,4 @@
   },
   "vision_end_token_id": 248054,
   "vision_start_token_id": 248053
-}
+}

model.safetensors.index.json

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:483170bd31d6eebd8e322b1742dfe0435791105bff74846a8f158201037b1f56
-size 13071821
+oid sha256:06334b7b33fc689c769c71f5ad278423aa92c09f0bcd28d9249b88d21e971215
+size 13127327

mtp.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e6c8f36002c1db3906a771f613a7295486402f5c39e782df1607f01bce5f2e3f
+size 1689376064