Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4-MTP

NVFP4 quantisation of nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451, built to run on a 128 GB DGX Spark, with the MTP speculative-decoding head intact.

This is the same quantisation as PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4 plus one 105 MB tensor that the other repo is missing. If you want speculative decoding, use this one. If you do not, either works.

	this repo (-MTP)	the plain -NVFP4 repo
tensors	2399	2398
mtp.fc.weight	present	absent
runs --speculative-config method=mtp	yes	loads, then drafts garbage
size	20.6 GB	20.5 GB

---

Read this first: the missing tensor problem

The upstream bf16 repo was re-uploaded on 2026-07-21 to add mtp.fc.weight - the fusion projection the MTP head needs to combine its two inputs. A checkpoint pulled before that date has mtp.layers.0.* but no mtp.fc, and this is a silent failure.

vLLM disables strict weight-initialisation checking for quantized configs. The missing tensor is allocated, nothing is loaded into it, and it keeps whatever was in that memory. The MTP head then drafts from a random projection: no exception, no warning, nothing in the log. Every drafted token is rejected, you pay the drafting cost for nothing, and decode gets slower. The only symptom is a number you were probably not measuring.

Check your own copy:

import json, urllib.request
idx = json.load(urllib.request.urlopen(
    "https://huggingface.co/<repo>/resolve/main/model.safetensors.index.json"))
print("mtp.fc.weight:", "mtp.fc.weight" in idx["weight_map"])

False means MTP will not work for you, however healthy it looks.

Serving it

Tested on vLLM 0.25.2.dev0 (arm64/CUDA 13 build), DGX Spark GB10, sm_121a.

Plain, no speculative decoding:

vllm serve PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4-MTP \
  --kv-cache-dtype fp8 --attention-backend flashinfer \
  --max-model-len 131072 --gpu-memory-utilization 0.72 --max-num-seqs 64 \
  --reasoning-parser qwen3 --trust-remote-code \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml

With MTP (recommended): add

  --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

No separate draft model - vLLM rewrites this checkpoint's own config into a Qwen3_5MTP draft and reads the head out of these weights. It is a reasoning model, so pass chat_template_kwargs: {"enable_thinking": false} if you do not want the budget spent inside <think>.

With DFlash (much faster on file-editing work, much slower on prose, and it needs a patched draft because the published one does not load on vLLM at all): see the recipe repo linked below.

---

Is speculative decoding worth it? Measured, not guessed

540 measurements on this checkpoint: 3 configs x 3 workloads x 10 concurrency levels x 6 shuffled rounds, all clean, standard deviation mostly under 1%.

The plain-English version

An AI writes one word at a time, and each word means re-reading the whole model from memory. That memory trip is the slow part. Speculative decoding adds a small fast helper that guesses the next few words so the big model can check them in one go. Right guesses are free. Wrong guesses are thrown away and cost you time.

So it is a bet on whether the next words are predictable.

• MTP guesses 2 words ahead and is usually right. Small bet, reliable.
• DFlash guesses 15 words ahead at once. Astonishing when the text is predictable; wasteful when it is not.

Text is predictable when the model is mostly copying something you just gave it - "here is my file, rename this variable" - which is what coding assistants do all day. It is unpredictable when the model is writing something new.

How often each helper guessed right, on this model:

	writing prose	writing code	editing a file
MTP	47%	76%	100%
DFlash	6%	19%	83%

DFlash guessing right 6% of the time on prose is the whole reason it is not a magic speed-up button.

Rank the workloads by that hit rate and you have ranked them by outcome, for both methods, at every concurrency level. It is not really about concurrency.

MTP is a safe upgrade. Use it by default.

Aggregate tokens/sec vs no speculative decoding:

workload	MTP vs no-spec
code	wins at every concurrency level tested, up to +70%
edit	wins at every level, up to +104%
prose	wins at 1-8, 24, 32 streams (up to +32%); loses at 12 (-11%), 16 (-10%), 64 (-7%)

It costs about 10% of the KV cache pool and nothing else. On code and editing there is no concurrency at which it is the wrong choice. On free-form prose above 8 concurrent streams it is roughly a coin flip - that is where its acceptance is weakest (47%).

Writing code - MTP (orange) leads base (blue) at every level:

Editing files - DFlash (green) is in another league, MTP still well clear of base:

Free-form prose - DFlash drops below doing nothing from 4 streams up:

DFlash is a specialist tool with a sharp edge

Single stream on editing work it is +716% over no speculation and +311% over MTP. That is real. But:

• On prose it is beaten by doing nothing from 4 concurrent streams up.
• It costs 40% of the KV pool (second model + 15 draft slots per stream).
• It caps at about 22 concurrent streams on this hardware and queues the rest - median time to first token at 64 streams goes to 138 seconds.

Ask for 64 streams and count how many actually run at once. Base and MTP track the diagonal; DFlash flattens at 22, so its throughput lines beyond that point are a capacity limit rather than a speed result:

Rule of thumb: DFlash below its ~22-stream ceiling for edit-heavy agentic coding; MTP for everything else and for crowds.

Full charts, tables, the 540-cell dataset and the fix that makes DFlash load on vLLM at all:

-> Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4_Dflash_DGX_recipe

A warning about averages

Averaging prose/code/edit assumes editing is exactly a third of your traffic. 72% of DFlash's average comes from the edit workload alone. Drop editing and average only prose and code, and MTP wins from 4 streams up. Measure your own mix instead - acceptance rate tells you directly:

curl -s http://127.0.0.1:8000/metrics \
  | grep -E 'spec_decode_num_(accepted|draft)_tokens_total'

Near 0.06 is prose-like (do not use DFlash); near 0.83 is edit-like (do).

---

How this was made

Source: nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451 - bf16, 1199 tensors, 55,457,998,304 bytes, a nuslerp mergekit merge of two Qwen3.6-27B Architect-Polaris Fable variants. Uploaded with nightmedia's permission.

Quantised with NVIDIA ModelOpt 0.45.0, examples/llm_ptq/hf_ptq.py:

--qformat nvfp4 --calib_size 512 --dataset cnn_dailymail
--batch_size 0 --inference_tensor_parallel 1

ModelOpt auto-excludes linear_attn, the vision tower, mtp.*, embeddings and lm_head from quantisation - see quant_summary.txt in this repo for the per-layer record, and hf_quant_config.json for the exclusion patterns. Output: 2398 tensors, 20,487,991,904 bytes.

mtp.fc.weight ([5120, 10240] BF16, 104,857,600 bytes) was then taken from the corrected upstream revision and added as model-mtp-fc.safetensors, with the index updated - 2399 tensors, 20,592,849,504 bytes. vLLM's own source expects this tensor to be unquantized in NVFP4 checkpoints (qwen3_5_mtp.py: "mtp.fc is stored as BF16 in NVFP4 checkpoints ... Force unquantized"), so a spliced BF16 tensor is exactly what a fresh PTQ run would have emitted.

The three VL processor configs (preprocessor_config.json, processor_config.json, video_preprocessor_config.json) are included - ModelOpt does not emit them and vLLM hard-fails a VL model at init without them.

Limitations

• Measured on one machine, one model. DGX Spark GB10, 121.7 GiB unified memory, sm_121a, arm64, a community arm64/CUDA-13 vLLM build. No claim about other hardware, multi-node, or other quantizations.
• Acceptance rates are specific to this checkpoint. The MTP head was inherited through a four-generation mergekit merge and never retrained. A different model will have different acceptance, and therefore different crossover points.
• No quality evaluation was run here. NVFP4 is a lossy 4-bit quantisation. These numbers are throughput only; nothing in this card says the output is as good as the bf16 source.
• Benchmarks ran with thinking disabled and 512-token generations. Long reasoning traces were not measured.
• min_p and logit_bias do not work under speculative decoding (vLLM warns at startup).

Credits

• nightmedia - the bf16 merge this is quantised from, and its Architect-Polaris merge stages. All the model work is theirs; uploaded with their permission. That merge in turn credits Qwen/Qwen3.6-27B and its own upstreams.
• z-lab - the DFlash draft used in the comparison.
• NVIDIA ModelOpt - the PTQ pipeline.
• vLLM - the serving engine, including the Qwen3_5MTP support that makes the MTP head usable at all.