AEON-7/Qwen3.6-35B-A3B-heretic-NVFP4

Qwen3.6-35B-A3B-heretic — NVFP4 (v2 multimodal-preserved)

✅ Validated on the unified AEON vLLM Ultimate image ghcr.io/aeon-7/aeon-vllm-ultimate:latest (= :2026-06-18-v0.23.0-dflashfix; rollback :2026-06-11-pr41703) — vLLM 0.23.0 built from source for GB10 / sm_121a + the AEON DFlash stack. Loads + serves cleanly (compressed-tensors MoE) — including working multimodal image understanding (a vision tensor-prefix bug was fixed 2026-06-18 — see the Vision fix section below) — and with DFlash num_speculative_tokens: 11 now scales to c=64 with up to ~740 tok/s aggregate and no concurrency crash. This single image supersedes the older vllm-spark-omni-q36:v1.2 container. Recommended container base.

🚀 PRODUCTION DEPLOYMENT GUIDE: github.com/AEON-7/Qwen3.6-35B-A3B-heretic-NVFP4-DFlash

The GitHub repo is the definitive turn-key setup for DGX Spark — end-to-end deployment guide, validated OpenClaw config, and a concurrency-sweep benchmark harness.

• Image: ghcr.io/aeon-7/aeon-vllm-ultimate:latest (vLLM 0.23.0 source-built for sm_121a + the AEON speculative-decoding stack — see What we fixed for the DGX Spark)
• Pairs with z-lab/Qwen3.6-35B-A3B-DFlash drafter (must be post 2026-04-19 revision)
• Production-stable under sustained chat load and concurrent agent fleets up to c=64 on DGX Spark

What changed in v2 (2026-04-19)

v1 of this checkpoint had model.language_model.layers.X.* keys remapped to model.layers.X.* so vLLM's text-only Qwen3_5MoeForCausalLM loader would pick them up. That layout was unstable in production — intermittent NaN/crash in the prefix-strip codepath during real chat sessions.

v2 re-quantizes the same source (tvall43/Qwen3.6-35B-A3B-heretic) with AutoModelForImageTextToText, restoring the canonical multimodal architecture (the text path):

• Architecture: Qwen3_5MoeForConditionalGeneration (vLLM's canonical class — no registry hack required)
• Keys: model.language_model.layers.X.* retained natively (no post-quantization key rewriting)
• 27-block ViT vision encoder preserved BF16 — but note its tensors were mis-nested as model.language_model.visual.* and silently skip-loaded until the 2026-06-18 vision fix below; image inputs did not work in v2 until that rename
• 30 linear-attention (Mamba/GDN) layers preserved BF16
• All 122,880 per-expert NVFP4 keys (40 layers × 256 experts × 3 projections × 4 quant components)

vLLM serves it via the canonical multimodal class with no prefix-strip code path in the inference hot loop. Result: rock-solid text stability where v1 was crashing on virtually every interaction. (Vision, however, was not yet functional in v2 — the ViT was mis-nested and skip-loaded; see the Vision fix section below, which corrected this on 2026-06-18.)

⚠️ If you cloned v1 of this repo, delete and re-pull. Same URL — v2 commits replaced v1.

Vision fix (2026-06-18) — image inputs now work

v2 (above) preserved the vision tower's BF16 weights, but a quantization-time ignore-regex defect nested the 27-block ViT one level too deep: the 333 vision tensors were written as model.language_model.visual.* instead of the sibling model.visual.* that vLLM's Qwen3_5MoeForConditionalGeneration expects (it builds self.visual at model.visual — a sibling of self.language_model, not a child). vLLM therefore silently skip-loaded the entire vision tower (WARNING … Parameter visual.* not found in params_dict, skip loading) and ran it uninitialized — text was perfect, but any image input produced !!!! garbage.

This revision renames those 333 vision tensors to model.visual.* — a header-only edit, so the NVFP4/BF16 weight data is byte-for-byte identical (no re-quantization; same file size, same data offsets). vLLM now loads the full ViT (0 skip-loads) and image understanding works, validated on aeon-vllm-ultimate:latest: a shapes/colors probe scored 7/7 (all shapes + colors + on-image text) and a richer scene 5/7, with LLM text unchanged. The AEON-7 27B multimodal siblings already used the correct model.visual.* layout and were unaffected by this defect.

If you pulled this repo before 2026-06-18 and saw !!!! on image prompts, re-pull — the corrected model.safetensors is live at the same URL (weight data identical; only the vision tensor names changed).

---

NVFP4-quantized version of tvall43/Qwen3.6-35B-A3B-heretic — an abliterated (decensored, 5/100 refusal rate) Qwen 3.6 35B-A3B Mixture-of-Experts multimodal model with thinking/reasoning capabilities.

Quantized using llmcompressor with the compressed-tensors nvfp4-pack-quantized format. Calibrated with 256 samples from open-platypus over 40 sequential decoder-layer stages. Vision encoder, linear-attention (Mamba/GDN) layers, MoE routers, gates, norms, and lm_head/embed_tokens preserved in BF16.

Designed for deployment on NVIDIA DGX Spark (GB10, Blackwell SM 12.0+) with native FP4 tensor-core support. Pairs with z-lab/Qwen3.6-35B-A3B-DFlash for spec-decode acceptance of 2.7-4.4 mean accepted tokens per target step on greedy workloads.

---

🚀 Quickstart (copy-paste)

Complete recipe for a DGX Spark / Blackwell box: pull the container, pull this NVFP4 model, pull the DFlash drafter (fresh), then serve with the vetted flags + DFlash spec-decode. No --mamba-block-size is needed for this 35B-A3B body.

# 1. Pull the unified AEON vLLM image (anonymous public GHCR pull)
docker pull ghcr.io/aeon-7/aeon-vllm-ultimate:latest

# 2. Pull THIS NVFP4 model (fresh — v2 commits replaced v1 at the same URL)
huggingface-cli download AEON-7/Qwen3.6-35B-A3B-heretic-NVFP4 --local-dir ./aeon-model

# 3. Pull the DFlash drafter, fresh (must be a post 2026-04-19 revision)
huggingface-cli download z-lab/Qwen3.6-35B-A3B-DFlash --local-dir ./aeon-drafter

# 4. Serve (NVFP4 body + DFlash drafter, n=11)
docker run --gpus all --ipc host --network host \
  -e VLLM_TEST_FORCE_FP8_MARLIN=1 \
  -v ./aeon-model:/model:ro \
  -v ./aeon-drafter:/drafter:ro \
  --entrypoint vllm \
  ghcr.io/aeon-7/aeon-vllm-ultimate:latest \
  serve /model \
  --quantization compressed-tensors \
  --attention-backend flash_attn \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --enable-auto-tool-choice \
  --gpu-memory-utilization 0.65 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --trust-remote-code \
  --max-model-len 40960 \
  --max-num-seqs 64 \
  --max-num-batched-tokens 32768 \
  --speculative-config '{"method":"dflash","model":"/drafter","num_speculative_tokens":11}'

Required with DFlash: --max-num-seqs and --max-num-batched-tokens must be set (as above) — without them the speculative-decoding scheduler can compute a negative token budget and the server fails to boot (max_num_scheduled_tokens is set to -512…). Raise --max-model-len (e.g. 262144) for full long-context once running; see the production compose below.

Lower --gpu-memory-utilization to 0.70 if you co-locate ASR/TTS/embedding services on the Spark's unified memory — on GB10 keep it 0.6-0.7, since above ~0.8 the shared CPU+GPU pool page-thrashes and stalls the box (even 0.85 stalls); go lower for co-located services, high concurrency, fp16 KV, or DFlash. For the full production setup (docker-compose, served-model aliases, 256K --max-model-len, systemd, OpenClaw integration) see the Quick Start (DGX Spark) and Production docker-compose sections below.

---

Performance Benchmarks

v0.23.0 build — current production image

These are the current measured numbers on ghcr.io/aeon-7/aeon-vllm-ultimate:latest (vLLM 0.23.0 + AEON sm_121a build + DFlash num_speculative_tokens: 11 via z-lab/Qwen3.6-35B-A3B-DFlash), on NVIDIA DGX Spark (GB10, sm_121a, 128 GB unified memory).

The unified image's specific win on this 35B-A3B MoE is clean concurrency scaling to c=64 with no crash (the pre-fix image crashed under concurrent speculative decoding at c≥32 — see What we fixed for the DGX Spark), plus long-context DFlash acceptance that holds ~42–58% out to 32K-token prompts. Aggregate throughput rises from 75–124 tok/s single-stream to **430–740 tok/s at c=64 depending on category** — DFlash-acceptance-driven: ~610–740 tok/s on structured code/math/reasoning/extraction (which draft well) and ~430–475 tok/s on creative prose/natural-language (lower draft acceptance).

Single-stream (c=1), by category

Greedy (T=0), 256-token output, mixed-domain prompt set. decode_tok_s_p50 / ttft_p50_ms / tpot_p50_ms / prefill (pp_tok_s_p50) / DFlash position-0 acceptance.

Category	🟢 Decode tok/s	TTFT p50	TPOT p50	Prefill (PP)	DFlash accept
Coding	91.7	88 ms	10.9 ms	509 tok/s	32%
Math	123.6	113 ms	8.1 ms	494 tok/s	48%
Reasoning	120.6	120 ms	8.3 ms	359 tok/s	46%
Prose	75.2	137 ms	13.3 ms	234 tok/s	24%
Natural language	91.8	104 ms	10.9 ms	326 tok/s	32%
Extraction / JSON	79.8	103 ms	12.5 ms	468 tok/s	28%

Aggregate throughput vs concurrency (c=1 → c=64)

Aggregate tok/s across all active streams (median of the per-category levels), 256-token output:

Concurrency	Coding	Math	Reasoning	Prose	Natural language	Extraction / JSON
1	89	118	115	73	89	78
8	360	416	388	252	296	399
16	464	559	518	362	399	557
32	569	715	660	435	495	685
64	609	740	669	429	474	716

Scales cleanly through c=64 with zero errors at every level (the pre-fix image died at c≥32 under speculative decoding). Aggregate is compute-bound on this 35B-active-3B MoE — per-request decode falls as streams divide the GPU, but no stream stalls or crashes.

Re-validated 2026-06-19 on the exact published recipe (a fresh pull of aeon-vllm-ultimate:latest + the corrected weights, DFlash n=11): multimodal vision works end-to-end (7/7 on an image probe, 0 vision skip-loads) and the throughput spread reproduced with zero errors at c=64 — structured categories 705–781 tok/s @ c=64 (peak ~800 tok/s @ c=32, Reasoning), creative-text 430–474 tok/s @ c=64; DFlash acceptance 46–55% (structured) / ~22–27% (creative) at short context, holding **40–45% at 16–20K-token** context.

Long-context tiers (DFlash acceptance holds)

Measured at ~16K and ~32K input tokens (median prompt_tokens_p50), greedy, 256-token output. The point of the AEON DFlash patches (PR #40898 SWA + PR #41703 prefix-cache safety) is that draft acceptance does not collapse as context grows:

Context tier	Measured prompt (p50)	Decode tok/s (c=1)	TTFT (c=1)	DFlash accept
~16K	16.2K – 20.3K	90.8 – 106.1	3.3 – 4.1 s	41 – 52%
~32K	32.6K – 40.8K	73.0 – 94.4	7.1 – 9.5 s	43 – 58%

Acceptance at 32K (43–58%) is on par with — and on the Extraction/JSON class, above — short-context acceptance, confirming the drafter's sliding-window layers are running as true SWA rather than collapsing past ~2K tokens.

Stock baseline note: there is no stock vanilla-vLLM baseline for this 35B-A3B checkpoint yet. A fresh fully-vanilla re-benchmark (default vLLM, no DFlash, no AEON/sm_121a optimizations) is pending; when it lands, a stock-vs-optimized contrast will be added here. The figures above are all measured on the optimized aeon-vllm-ultimate:latest (vLLM 0.23.0) build. The earlier v1.2-image figures retained below for reference are likewise optimized-path numbers (different image, DFlash k=15), not a vanilla baseline.

⚠️ DFlash speedup is workload-dependent. Greedy reasoning workloads (math, code, extraction) hit the highest acceptance (≥40%); creative / open-ended / sampled prose is lower (~24%) and more variable. Use T=0 for maximum DFlash speedup.

---

Earlier vllm-spark-omni-q36:v1.2 reference (DFlash k=15)

The detailed single-stream / concurrent tables below were measured on the earlier ghcr.io/aeon-7/vllm-spark-omni-q36:v1.2 image at DFlash k=15, not on the current aeon-vllm-ultimate:latest at n=11. They are retained as a representative reference for the Spark DFlash path; the current production numbers are the v0.23.0 section above. The unified image's specific gain is clean c=64 concurrency scaling and held long-context acceptance, not these short-context single-stream tok/s.

1. Single-Stream Performance

Best for interactive chat and agentic UX. All measurements greedy (T=0) unless noted.

Decode rate (10 trials, 200-token outputs)

Statistic	tok/s
Median	83.9
p95	127.5
Min	41.1
Max	127.5

Variance reflects DFlash acceptance differences across prompt classes — math/code prompts hit ~125 tok/s with high drafter agreement, more open-ended prompts settle around 60-90 tok/s.

TTFT by prompt length (5 trials per class)

Prompt class	Approx. input tokens	TTFT p50	TTFT p95	TTFT min	Effective prefill
Tiny	2	99 ms	102 ms	98 ms	20 tok/s
Short	7	114 ms	115 ms	110 ms	62 tok/s
Medium	50	123 ms	128 ms	121 ms	407 tok/s
Long	465	259 ms	314 ms	257 ms	1,797 tok/s

Sub-130ms TTFT for any prompt under ~50 tokens — fixed kernel-launch overhead dominates short prefill.

Decode rate by output length (3 trials per length)

Max tokens	Actual tokens (median)	TTFT	Decode rate	Total latency
50	50	113 ms	70.1 tok/s	0.82 s
200	200	112 ms	88.4 tok/s	2.37 s
500	331*	116 ms	115.6 tok/s	4.44 s
1000	330*	113 ms	118.3 tok/s	6.28 s

* model emitted EOS naturally before hitting max_tokens.

Decode rate increases with output length — DFlash steady-state amortization improves over the first 100-200 tokens once the drafter and target lock into a stable acceptance pattern.

Sampling: greedy vs stochastic (5 trials per mode)

Mode	Decode p50	Decode p95	TTFT p50
Greedy (T=0)	76.5 tok/s	123.0 tok/s	115 ms
Stochastic (T=0.7)	64.8 tok/s	125.4 tok/s	113 ms

15% degradation T=0 → T=0.7. Less dramatic than typical for spec-decode systems — DFlash's drafter remains useful even at moderate sampling. Use T=0 for max DFlash speedup; T=0.7 for diversity.

Long-prompt prefill (RAG / document workloads)

Input tokens	TTFT (≈ prefill)	Prefill rate	Decode rate after prefill
1K	519 ms	1,973 tok/s	48.8 tok/s
4K	2,594 ms	1,579 tok/s	41.1 tok/s
16K	8,007 ms	2,046 tok/s	34.6 tok/s
32K	19,368 ms	1,692 tok/s	23.0 tok/s

Prefill rate plateaus around 2K tok/s due to (a) the drafter prefilling the same context in parallel and (b) Qwen3.6's 30 linear-attention (Mamba/GDN) layers having higher prefill constant factor than parallel softmax attention. Decode-after-prefill drops gracefully (~50% from 1K → 32K context).

Single-stream summary

Metric	Value
Single-stream decode (200-tok output)	83.9 tok/s median
Decode @ 500-1000 tok output (DFlash steady state)	115-118 tok/s
Short-prompt TTFT	99-128 ms
16K-prompt TTFT	8.0 s
32K-prompt TTFT	19.4 s
Peak prefill throughput	~2,046 tok/s @ 16K prompt
Decode rate with 32K context	23.0 tok/s (53% drop vs short context)

---

2. Concurrent-Session Performance

Best for agent fleets and multi-user serving. 3 trials per level, median run reported (sorted by aggregate throughput). Mixed prompts, 200-token output, T=0.7 (stochastic — production-realistic), SSE streaming.

Throughput scaling (N concurrent clients, 200-tok output)

Concurrent	Errors	Agg tok/s (median of 3)	Per-req decode p50	Per-req decode min	TTFT p50	TTFT p95
1	0	102.9	109.1	109.1	111 ms	111 ms
2	0	131.3	94.0	68.9	144 ms	144 ms
4	0	128.1	48.5	38.9	191 ms	191 ms
8	0	163.3	29.2	14.2	355 ms	356 ms
16	0	227.6	19.3	8.6	501 ms	503 ms
32	0	275.5	11.6	5.2	701 ms	703 ms
64	0	310.8	6.9	3.3	1.07 s	11.2 s
128	0	313.6	6.5	3.0	14.1 s	46.7 s

Zero errors across all 384 requests in the concurrent sweep (3 runs × 128-conc top level alone = 384, plus all lower levels = 1,200+ total).

Aggregate throughput plateaus at ~313 tok/s from 64 concurrent onward — that's the GPU's compute wall on this 35B-active-3B MoE with linear-attention KV reads + DFlash drafter overhead. TTFT spikes severely at 128 concurrent (14s p50, 47s p95) because all 128 sequences fit in the scheduler but compute is fully saturated, so each token's worth of work is divided across 128 streams. For latency-sensitive UX, target 16-32 concurrent; for max throughput, use the full 128.

TTFT-only scaling (1-token output, prefill + first-token)

Measures pure scheduler queue contention — critical for agent UX:

Concurrent	TTFT p50	TTFT p95	TTFT min	TTFT max
1	74 ms	75 ms	72 ms	75 ms
4	99 ms	100 ms	97 ms	100 ms
16	249 ms	263 ms	238 ms	263 ms
64	560 ms	698 ms	451 ms	707 ms

TTFT stays sub-700ms through 64 concurrent — smooth UX for small agent fleets. Beyond 64, TTFT accumulates queue-wait time as compute is fully consumed.

Concurrent with 1K-token prompts (RAG-style workload)

50-token output with 1,024-token prompts — simulates agents doing document QA or retrieval-augmented responses. Median of 2 runs.

Concurrent	Errors	Agg tok/s	TTFT p50	TTFT p95	Decode p50
1	0	23.1	494 ms	494 ms	44.1
4	0	39.5	1,673 ms	1,720 ms	24.6
16	0	47.1	6,179 ms	6,180 ms	10.6
64	0	49.8	19,297 ms	33,352 ms	2.5

RAG throughput peaks around 50 tok/s at 16-64 concurrent. The aggregate is lower than the short-prompt sweep because each request spends most of its wall-clock in prefill (1K tokens) rather than decode. Use prefix caching if your RAG workload has repeated context blocks — the production compose enables --enable-prefix-caching which can give 5-10× speedup on shared-prefix RAG.

Concurrent-session summary

Metric	Value
Peak aggregate throughput	313.6 tok/s @ 128 concurrent (median of 3 trials)
Scaling from 1 → 128	3.05× throughput (compute-bound — DFlash + 35B MoE saturates GB10 around 64 streams)
Per-request decode @ 128	6.5 tok/s p50, 3.0 min
TTFT @ 64 concurrent	1.07 s p50 (acceptable for agent fleets)
TTFT @ 128 concurrent	14.1 s p50 (queue-bound — useful for batch only)
Error rate across full bench	0.0% (1,200+ requests, conc 1 → 128)
Best concurrency for chat UX	4-16 (per-req 19-48 tok/s, TTFT < 500 ms)
Best concurrency for max throughput	64-128 (saturated compute, TTFT trade-off)

---

Key Performance Metrics Summary

Metric	Value
Single-stream decode (200-tok output)	83.9 tok/s median
Single-stream decode @ DFlash steady state	118 tok/s (1000-tok output)
Short-prompt TTFT	99-128 ms
Peak aggregate throughput	313.6 tok/s @ 128 concurrent
TTFT @ 16 concurrent (smooth UX)	501 ms p50
TTFT @ 64 concurrent (still usable)	1.07 s p50
Greedy vs stochastic decode penalty	15% (76.5 → 64.8 tok/s)
DFlash position-0 acceptance (greedy workloads)	62-78%
Mean accepted tokens per target step	2.7-4.4
Long-context decode @ 32K prompt	23.0 tok/s
Total bench wall-clock	11 minutes (1,200+ requests, 0 errors)

---

Scaling efficiency (200-tok concurrent test)

Concurrency	Throughput gain vs 1-req
1	1.0×
4	1.2×
16	2.2×
64	3.0×
128	3.05×

Scaling is GPU-compute-bound rather than memory-bound — DFlash on a 35B MoE with hybrid linear+full attention saturates the GB10's compute around 64 concurrent. Per-request throughput degrades from 109 tok/s (1-req) to 6.5 tok/s (128-req). For comparison, a non-spec-decode setup would scale much more linearly but lose the ~2-4× single-stream speedup DFlash provides.

---

Test methodology notes

• enable_thinking=false — bench disables Qwen3.6's thinking tag for clean decode-rate measurement. Production with thinking on adds reasoning-token overhead before content emission (use max_tokens ≥ 2048 for thinking-enabled requests).
• DFlash speedup is workload-dependent — math, code, agentic, and reasoning workloads at T=0 hit the highest acceptance rates. Creative writing or open-ended chat sees lower acceptance.
• Mixed-prompt set in concurrent tests: code, math, QA, creative writing, single-line answers — to avoid biasing toward DFlash-friendly prompts.
• 3 trials per concurrency level for the throughput sweep, median run (by aggregate tok/s) reported. RAG section uses 2 trials.
• 200-token output as the standard test length (except TTFT-only test which uses 1 token, RAG which uses 50, and decode-by-output which sweeps 50→1000).
• Error tracking: 0/1,200+ requests failed across the full test (all sections combined).
• Reproducible: bench script at scripts/bench_full.py; raw JSON results at bench/qwen36_v2_2026-04-20.json.

---

⚠️ IMPORTANT REQUIREMENTS

#	Requirement	Why
1	Native Blackwell GPU (SM 10.0+ — B200, GB10, RTX PRO 6000 Blackwell, RTX 5090)	NVFP4 needs hardware FP4 tensor cores
2	vLLM with sm_121a NVFP4 kernels — use ghcr.io/aeon-7/aeon-vllm-ultimate:latest (vLLM 0.23.0 source-built for sm_121a + the AEON DFlash stack)	Stock vLLM wheels don't compile FP4 kernels for SM 12.x; the SM121 workarounds aren't all upstream yet
3	--quantization compressed-tensors (NOT modelopt)	This checkpoint uses llmcompressor's compressed-tensors NVFP4 format
4	--trust-remote-code	Qwen3.6 ships custom modeling code
5	--attention-backend flash_attn (when using DFlash)	DFlash spec decode requires flash_attn backend
6	VLLM_TEST_FORCE_FP8_MARLIN=1 env (defensive pin)	Pins NVFP4 MoE backend to MARLIN. As of 2026-04-21, vLLM's auto-selector arrives at MARLIN anyway because every other MoE backend rejects our 256-expert × 512-intermediate shape — this env defends against future vLLM versions adding a half-broken backend. The linear NVFP4 path is unaffected and uses native FP4 tensor cores (FlashInferCutlassNvFp4Linear).
7	DFlash drafter from post 2026-04-19 revision	Earlier z-lab drafter had a long-context crash bug
8	Latest transformers (≥5.5.4)	qwen3_5_moe model_type registration

---

Quick Start (DGX Spark, with DFlash spec decode)

# 1. Pull the image (anonymous public GHCR pull — anyone can run this)
docker pull ghcr.io/aeon-7/aeon-vllm-ultimate:latest

# 2. Pull both models
sudo mkdir -p /opt/qwen36 && sudo chown $USER:$USER /opt/qwen36
cd /opt/qwen36
export HF_HUB_ENABLE_HF_TRANSFER=1
hf download AEON-7/Qwen3.6-35B-A3B-heretic-NVFP4 --local-dir ./qwen36-nvfp4 &
hf download z-lab/Qwen3.6-35B-A3B-DFlash         --local-dir ./qwen36-dflash &
wait

# 3. Get the production compose file
curl -fsSL \
  https://raw.githubusercontent.com/AEON-7/Qwen3.6-35B-A3B-heretic-NVFP4-DFlash/main/examples/docker-compose.yml \
  -o docker-compose.yml

# 4. Start
docker compose up -d
docker compose logs -f   # wait for "Application startup complete" (~3-5 min)

# 5. Test (use temperature=0 + ≥2048 max_tokens for thinking-enabled requests)
curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen36-fast",
    "messages": [{"role":"user","content":"What is 17 × 23? Show your work."}],
    "max_tokens": 2048,
    "temperature": 0
  }'

Full step-by-step (with pre-flight checks, smoke tests, systemd service, OpenClaw integration): github.com/AEON-7/Qwen3.6-35B-A3B-heretic-NVFP4-DFlash/blob/main/docs/dgx-spark-setup.md

Production docker-compose (the actual flags that work)

services:
  vllm:
    image: ghcr.io/aeon-7/aeon-vllm-ultimate:latest   # = :2026-06-18-v0.23.0-dflashfix; rollback :2026-06-11-pr41703
    container_name: vllm-qwen36-heretic
    restart: unless-stopped
    network_mode: host
    # The image ENTRYPOINT is /bin/bash, so use entrypoint: vllm + command: serve ...
    entrypoint: vllm
    environment:
      - VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
      - TORCH_CUDA_ARCH_LIST=12.1a
      - ENABLE_NVFP4_SM100=0
      - PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
      - NVIDIA_FORWARD_COMPAT=1
      - VLLM_TEST_FORCE_FP8_MARLIN=1     # pins the NVFP4 MoE backend to MARLIN (256×512 expert shape)
    volumes:
      - /opt/qwen36/qwen36-nvfp4:/models/qwen36
      - /opt/qwen36/qwen36-dflash:/models/qwen36-dflash
    command:
      - serve
      - /models/qwen36
      - --served-model-name
      - qwen36-35b-heretic
      - qwen36-fast
      - qwen36-deep
      - --host
      - 0.0.0.0
      - --port
      - "8000"
      - --tensor-parallel-size
      - "1"
      - --dtype
      - auto
      - --quantization
      - compressed-tensors
      - --max-model-len
      - "262144"
      - --max-num-seqs
      - "64"
      - --max-num-batched-tokens
      - "32768"
      - --gpu-memory-utilization
      - "0.70"
      - --enable-chunked-prefill
      - --enable-prefix-caching
      - --trust-remote-code
      - --enable-auto-tool-choice
      - --tool-call-parser
      - qwen3_coder
      - --reasoning-parser
      - qwen3
      - --speculative-config
      - '{"method":"dflash","model":"/models/qwen36-dflash","num_speculative_tokens":11}'
      - --attention-backend
      - flash_attn
    ipc: host
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Recipe notes (current build):

• --quantization compressed-tensors (NOT modelopt) — this checkpoint uses llmcompressor's NVFP4 format.
• --attention-backend flash_attn is required for the DFlash drafter on the body.
• num_speculative_tokens: 11 is the validated production default for this 35B-A3B body — acceptance and long-context hold peak around n≈10–11, above which they fall off. Do not set --kv-cache-dtype (the non-causal DFlash drafter requires BF16 KV).
• No --mamba-block-size is needed on the v0.23.0 image (block_size is now an int upstream; the prior patch_kv_cache_utils carry was dropped).
• --enforce-eager is NOT required — CUDA graphs run cleanly with the unified image's spec-decode CUDA-graph capture-size alignment patch + the post-2026-04-19 DFlash drafter. Cudagraphs on give ~30% over eager.
• --gpu-memory-utilization 0.70 leaves headroom for co-located ASR/TTS/embedding services on the Spark's unified memory; keep it 0.6-0.7 even when the LLM is the only major GPU service — above ~0.8 the shared CPU+GPU pool page-thrashes and stalls the box (even 0.85 stalls). Go lower for co-located services, high concurrency, fp16 KV, or DFlash.

---

What we fixed for the DGX Spark

All AEON models run on one unified container — ghcr.io/aeon-7/aeon-vllm-ultimate:latest (= :2026-06-18-v0.23.0-dflashfix; rollback :2026-06-11-pr41703) — vLLM v0.23.0 built from source for GB10 / sm_121a and merged with the AEON speculative-decoding stack.

Fix	What it does	Why it matters on GB10
DFlash high-concurrency fix (new)	Slices the speculative drafter's KV block-table to the unpadded batch (block_table[:num_reqs])	The drafter previously crashed at ≥32 concurrent requests (padded-vs-unpadded block-table shape mismatch in FlashAttention). Now scales cleanly to c=64. A port of upstream PR #43982, which fixed this for MTP but never for DFlash — present and unfixed even in the prior image.
Triton NVFP4 KV cache (PR #44389)	Software NVFP4 KV-cache path	The only 4-bit KV path on sm_121a (upstream's is hard-gated to B200) → ~3× KV capacity / longer context per GB of unified memory.
DFlash sliding-window attention (PR #40898)	Runs the drafter's SWA layers as true sliding-window	Long-context draft acceptance holds as agent histories grow (43–58% out to 32K tokens here) instead of collapsing past ~2k tokens.
Prefix-cache-safe DFlash (PR #41703)	Makes --enable-prefix-caching corruption-immune with the DFlash drafter	Lets shared-prefix agent/RAG workloads use prefix caching without garbled output under speculative decoding.
sm_121a-native build	TORCH_CUDA_ARCH_LIST=12.1a, ENABLE_NVFP4_SM100=0	Compiles the SM120-family CUTLASS NVFP4/FP8 kernels GB10 actually dispatches to — true 4-bit tensor-core throughput, no dead B200-only kernels.
sm_121a boot + CUDA-graph patches	RTLD-lazy _C_stable_libtorch load; spec-decode CUDA-graph capture-size alignment	Boots past MXFP4 (SM100-only) symbols absent on GB10; prevents cudaErrorIllegalAddress on partial-acceptance decode steps under speculative decoding.
Unified-memory tuning	--gpu-memory-utilization ≤0.70, FULL CUDA graphs, async scheduling, z-lab DFlash drafter	GB10 shares one LPDDR5X pool across CPU + GPU; conservative KV headroom avoids page-thrash while keeping FULL-graph + speculative-decode throughput.

The result

• Scales to 64 concurrent requests with no crash — the prior image crashed at c≥32 under speculative decoding (the block-table fix above). Aggregate throughput rises to ~430–740 tok/s at c=64 depending on category (acceptance-driven: ~610–740 on structured text, ~430–475 on creative prose/natural-language), zero errors at every level.
• Native NVFP4 4-bit compute on Blackwell tensor cores — the speed of 4-bit with near-16-bit accuracy.
• Speculative decoding (DFlash) holds high draft acceptance from short prompts (32–48% short-context) through long 16K–32K agent histories (43–58% at 32K), rather than collapsing past ~2K tokens.
• No measured single-stream speedup vs a stock vanilla baseline yet — there is no stock baseline for this checkpoint; a fresh fully-vanilla re-benchmark is pending. The unified image's concrete, measured win here is the c=64 concurrency scaling and the held long-context acceptance described above.

---

Model Architecture

Property	Value
Architecture	qwen3_5_moe (multimodal — Qwen3_5MoeForConditionalGeneration)
Total params	~35B
Active params	~3B / token
Layers	40 (3× Gated DeltaNet + 1× Gated Attention, repeating ×10)
Hidden	2048
Experts	256 routed + 1 shared, top-8 per token
Vocabulary	248,320
Native context	262,144 (256K)
Extended context (YaRN)	1,010,000 (1M+)
Multimodal	27-block ViT vision encoder (preserved BF16)

Hybrid Attention

Attention type	Layers	Q/K/V heads	Head dim
Gated DeltaNet (linear, BF16)	30 (3 of every 4)	QK 16, V 32	128
Gated Attention (NVFP4)	10 (1 of every 4)	Q 16, KV 2	256 (rotary 64)

---

Quantization Details

Parameter	Value
Tool	llmcompressor
Format	compressed-tensors nvfp4-pack-quantized
Scheme	NVFP4 (FP4 E2M1 + per-block FP8 e4m3 scales + per-tensor FP32 scales)
Block size	16
Calibration data	open-platypus (256 samples)
Calibration seq_len	2048
Pipeline	Sequential (Qwen3_5MoeDecoderLayer, layer-by-layer to GPU)
Hardware	NVIDIA RTX PRO 6000 Blackwell (96 GB)
Calibration wall-clock	~3 hours (40 decoder layers × ~3-4 min each)
Output	9 safetensors shards, ~22 GB total
Expert keys (NVFP4)	122,880 (40 × 256 × 3 × 4)
Visual keys (BF16)	~333
Linear-attn keys (BF16)	~270

Quantized layers (NVFP4)

• Gated Attention projections: q_proj, k_proj, v_proj, o_proj (10 layers)
• MoE experts (256 × 40 layers = 10,240 expert modules): gate_proj, up_proj, down_proj
• Shared expert: same projections

Excluded from quantization (kept BF16)

• lm_head, embed_tokens — accuracy-critical token projections
• *.mlp.gate, *.shared_expert_gate — MoE routing (sparsity-critical)
• *.norm.* — all RMSNorm layers
• *.visual.* — 27-block ViT vision tower
• *.linear_attn.* — 30 Gated DeltaNet (Mamba) layers (small relative to MoE; quantizing them tanks accuracy)

The exact recipe + script that produced this checkpoint is at scripts/qwen36_requant_v2.py.

---

Recommended sampling parameters

From the Qwen3.6 model card:

Mode	General	Coding	Math/Reasoning
Thinking	T=1.0, P=0.95, K=20, PP=1.5	T=0.6, P=0.95, K=20, PP=0.0	T=1.0, P=1.0, K=40, PP=2.0
Instruct (no think)	T=0.7, P=0.8, K=20, PP=1.5	—	T=1.0, P=0.95, K=20, PP=1.5

For maximum DFlash speedup: use T=0 (greedy). The drafter ↔ target agreement rate collapses with sampling — at T=0.7 you typically see ~10-20% acceptance vs. 60-78% at T=0.

The production compose registers 3 served-model aliases for the same backend so chat clients can route greedy vs sampled requests separately:

• qwen36-fast → intended for greedy/agentic (T=0)
• qwen36-deep → intended for creative/sampled (T=0.7)
• qwen36-35b-heretic → canonical name

Disable thinking per-request:

{"chat_template_kwargs": {"enable_thinking": false}}

Preserve thinking across multi-turn:

{"chat_template_kwargs": {"preserve_thinking": true}}

Common gotcha: with thinking enabled (default), Qwen3.6 spends most of its max_tokens budget on <think> reasoning before emitting content. Use max_tokens ≥ 2048 for thinking-enabled requests — lower budgets often produce content: null with finish_reason: "length".

---

Hardware Requirements

Tier	GPU	Notes
Target — production-validated	NVIDIA DGX Spark (128 GB unified, GB10 sm_121a)	Full 256K context, validated to c=64 concurrent on aeon-vllm-ultimate:latest
Compatible	RTX PRO 6000 Blackwell (96 GB)	What v2 was calibrated on. Same MoE-shape constraint applies (Marlin is still the only NVFP4 MoE backend that accepts 256×512 grouped GEMM regardless of chip); linear NVFP4 path uses CUTLASS native.
Compatible	B200 / GB200	Image rebuild required (SM 10.0, not SM 12.x)
Compatible	RTX 5090 (32 GB)	Reduced context, low concurrency
Minimum	Any Blackwell GPU (SM 10.0+)	Required for native FP4; sub-Blackwell can run via Marlin W4A16 fallback but with reduced throughput

---

Files

File	Size	Description
model-00001-of-00009.safetensors … model-00009-of-00009.safetensors	~22 GB total	NVFP4 quantized weights (~123,724 tensors across 9 shards)
model.safetensors.index.json	~5 MB	shard index
config.json	~7 KB	Model + quantization config (Qwen3_5MoeForConditionalGeneration)
tokenizer.json	~20 MB	Qwen tokenizer (248K vocab)
tokenizer_config.json	~1 KB
chat_template.jinja	~8 KB	Qwen3.6 chat template (thinking + tool calling)
preprocessor_config.json	~500 B	Image preprocessor (kept for multimodal compat)
generation_config.json	~213 B
recipe.yaml	~500 B	llmcompressor recipe used

---

Disclaimer

THIS IS AN UNCENSORED MODEL. By downloading, accessing, or using this model you expressly assume full and sole responsibility for all outputs generated, all actions taken based on outputs, and compliance with applicable laws. The authors are not responsible for any harmful, illegal, or objectionable content. These tools serve legitimate purposes including security research, red-teaming, content analysis, and creative work. Implement safeguards appropriate to your use case and jurisdiction.

License

Apache 2.0 (inherited from Qwen3.6 base).

Credits

• Base model: tvall43/Qwen3.6-35B-A3B-heretic — abliteration via Heretic v1.2.0
• Original target: Qwen/Qwen3.6-35B-A3B by Alibaba Tongyi
• DFlash drafter: z-lab/Qwen3.6-35B-A3B-DFlash — z-lab (Soroush Mohri et al.)
• Quantization tool: llmcompressor — Neural Magic / RedHat
• vLLM build chain: vllm-project/vllm v0.23.0 source-built for sm_121a + the AEON speculative-decoding stack (aeon-vllm-ultimate)
• SM121 stability investigation: rmagur1203/vllm-dgx-spark — independent 4-day, 144-config investigation that surfaced the Marlin requirement
• Quantized by: AEON-7 on NVIDIA RTX PRO 6000 Blackwell (RunPod)
• Production-validated on: NVIDIA DGX Spark (GB10)

---

☕ Support the work

If this release has been useful, tips are deeply appreciated — they go directly toward more compute, more models, and more open releases.

₿ Bitcoin (BTC) bc1q09xmzn00q4z3c5raene0f3pzn9d9pvawfm0py4

Ξ Ethereum (ETH) 0x1512667F6D61454ad531d2E45C0a5d1fd82D0500

◎ Solana (SOL) DgQsjHdAnT5PNLQTNpJdpLS3tYGpVcsHQCkpoiAKsw8t

ⓜ Monero (XMR) 836XrSKw4R76vNi3QPJ5Fa9ugcyvE2cWmKSPv3AhpTNNKvqP8v5ba9JRL4Vh7UnFNjDz3E2GXZDVVenu3rkZaNdUFhjAvgd

Ethereum L2s (Base, Arbitrum, Optimism, Polygon, etc.) and EVM-compatible tokens can be sent to the same Ethereum address.