nerkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill-GGUF

Qwen3.6-27B-DSV4Pro-Thinking-Distill — GGUF (native MTP + imatrix)

Lynn Agent default local model

This GGUF repo is the default local model recommendation for Lynn Agent. Lynn v0.85.6+ uses Qwen3.6-27B-DSV4Pro-Distill-MTP-Q5_K_M-imatrix.gguf as the first-choice edge model for 24GB+ VRAM / unified-memory machines; 32GB+ machines can choose the 35B-A3B Q5_K_M path, while 9B / 4B remain explicit low-config downgrades.

• Download Lynn Agent: GitHub Releases (current desktop/CLI release: v0.85.6)
• Default quant in Lynn: Q5_K_M imatrix + native MTP (--spec-draft-n-max 3)
• Recommended use: local agentic reasoning, coding, and cloud-independent fallback inside Lynn.

GGUF quantizations of nerkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill — a DeepSeek-V4-Pro ReAct-style reasoning distillation of Qwen3.6-27B (dense). Every GGUF carries a native MTP (multi-token-prediction) head and is imatrix-corrected (except Q8_0/BF16). Distillation only changes the thinking style (the model learns to converge), not coding or knowledge.

Quantizations

File	Size	Notes
…-MTP-IQ4_XS-imatrix.gguf	15.3 GB	16 GB-VRAM floor — fits 16 GB no-offload; smallest we ship
…-MTP-Q4_K_S-imatrix.gguf	15.8 GB	speed-first
…-MTP-Q4_K_M-imatrix.gguf	16.8 GB	recommended balance
…-MTP-Q5_K_M-imatrix.gguf	19.5 GB	higher quality
…-MTP-Q6_K-imatrix.gguf	22.4 GB	near-lossless
…-MTP-Q8_0.gguf	29.0 GB	virtually lossless
…-MTP-BF16.gguf	54.7 GB	full-precision reference
…-mmproj-F16.gguf	0.9 GB	vision projector (image input)

IQ4_XS is the floor. Sized for a 16 GB-VRAM card with zero CPU offload (14.3 GiB weights + headroom for KV). Smoke-verified converging (3/3 reasoning prompts correct, 0 length-runaway, EN + ZH) at 31 t/s single-stream with the native MTP head (--spec-type draft-mtp, accept 0.80, mean accept-len 3.4). This is the smallest tier we ship — below IQ4_XS a 27B reasoning distill starts to lose convergence, so we stop here.

📏 GB vs GiB — why a 15.3 GB file fits a "16 GB" GPU. Sizes in this table are decimal GB (10⁹ bytes). GPU VRAM is built in GiB (2³⁰ bytes) but marketed as "GB" — so a "16 GB" card is really 16 GiB ≈ 17.2 GB. IQ4_XS (15.3 GB = 14.3 GiB) therefore loads fully on a 16 GiB card with ~1.7 GiB left for KV cache — no CPU offload. Rule of thumb: a file runs no-offload when its GiB size < the card's nominal "GB" number (here 14.3 < 16 ✓).

Evaluation — distill vs base, same-spec GGUF, thinking-on

Both rows use the same quant + MTP; the only variable is "distilled or not". Do not cross-compare quant tiers (Q4 and Q5 use different harnesses).

Q4_K_M (archived harness):

Dimension	Distill	Base	Δ
GPQA-Diamond-198	80.81% (160/198)	73.7%	+7.1
MMLU-500 (5-shot)	91.8%	91.6%	+0.2
GPQA unconverged empties	0	14	−14
coding-100 (real sandbox)	86/100	83/100	+3

Q5_K_M (streaming harness · conc4 · timeout 1800 · max_tokens 32000, finish_reason logged):

Dimension	Distill	Base	Δ
GPQA-Diamond-198	81.82% (162/198)	68.69% (136/198)	+13.13pp
MMLU-500	90.0%	89.6%	+0.4
GPQA finish=stop (converged)	198/198	186/198
GPQA finish=length (hit 32K wall)	0	12

Hard reasoning jumps (GPQA +7~13pp), knowledge holds, and the distill learns to converge — the base hits the 32K wall on 12 questions (never answers); the distill converges on all 198.

MTP — native multi-token-prediction (single-stream speedup)

The GGUF has a native MTP head (mainline llama.cpp --spec-type draft-mtp, no external draft model). Measured single-stream on DGX Spark GB10, Q4_K_M:

--spec-draft-n-max	TPS	accept rate	accept len
bare (no MTP)	10.4	—	—
n-max=3 ⭐	26.8	0.72	3.16

2.3–2.6× single-stream speedup, lossless (greedy speculative only accepts the target-argmax token). Speculation is single-stream — concurrency degrades it, so use bare multi-concurrency for throughput.

Run

llama-server -m Qwen3.6-27B-DSV4Pro-Distill-MTP-Q4_K_M-imatrix.gguf \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  -c 8192 --jinja --host 127.0.0.1 --port 8080

Thinking model — use temp 0.6 / top_p 0.95 (greedy loops to death). For image input add --mmproj Qwen3.6-27B-DSV4Pro-Distill-mmproj-F16.gguf.

---

中文

nerkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill 的 GGUF 量化版 —— Qwen3.6-27B(dense)的 DeepSeek-V4-Pro ReAct 思维蒸馏版。每个档位都焊了原生 MTP 头,并做了 imatrix 校正(Q8_0/BF16 除外)。蒸馏只改思维方式(学会"收口"),coding 与知识不降。

量化档位

文件	大小	说明
…-MTP-IQ4_XS-imatrix.gguf	15.3 GB	16G 显存底线 —— 16G 不 offload 可跑,最小档
…-MTP-Q4_K_S-imatrix.gguf	15.8 GB	速度优先
…-MTP-Q4_K_M-imatrix.gguf	16.8 GB	平衡档 / 低显存回退
…-MTP-Q5_K_M-imatrix.gguf	19.5 GB	Lynn 默认推荐(24GB+)
…-MTP-Q6_K-imatrix.gguf	22.4 GB	接近无损
…-MTP-Q8_0.gguf	29.0 GB	几乎无损
…-MTP-BF16.gguf	54.7 GB	全精度参考
…-mmproj-F16.gguf	0.9 GB	视觉投影(图像输入)

IQ4_XS 是底线档。 为 16 GB 显存、不 offload 而做(14.3 GiB 权重 + KV 余量)。冒烟实测收口干净(3/3 推理题正确、0 长度跑飞、中英),单流 31 t/s + 原生 MTP 头(--spec-type draft-mtp,接受率 0.80、平均接受长 3.4)。这是我们发布的最小档 —— 27B 推理蒸馏低于 IQ4_XS 会开始丢收口,到此为止。

📏 GB vs GiB —— 为什么 15.3 GB 的文件能进"16 GB"显卡。 本表大小用十进制 GB(10⁹ 字节);显卡显存按 GiB(2³⁰ 字节) 造却标"GB",所以一张"16 GB"卡其实是 16 GiB ≈ 17.2 GB。于是 IQ4_XS(15.3 GB = 14.3 GiB)能完整载入 16 GiB 卡,剩 ~1.7 GiB 给 KV —— 不用 offload。经验法则:文件的 GiB 数 < 显卡标称"GB"数,就能不 offload 跑(这里 14.3 < 16 ✓)。

评测 —— 蒸馏 vs 原版,同规格 GGUF,thinking-on

两行同量化 + 同 MTP,唯一变量是"是否蒸馏"。禁止跨量化档比较(Q4 与 Q5 用了不同 harness)。

Q4_K_M(旧 harness):

维度	蒸馏	原版	Δ
GPQA-Diamond-198	80.81%(160/198)	73.7%	+7.1
MMLU-500(5-shot)	91.8%	91.6%	+0.2
GPQA 未收口空答	0	14	−14
coding-100(真实沙箱)	86/100	83/100	+3

Q5_K_M(流式 harness · 并发4 · timeout 1800 · max_tokens 32000,逐题记 finish_reason):

维度	蒸馏	原版	Δ
GPQA-Diamond-198	81.82%(162/198)	68.69%(136/198)	+13.13pp
MMLU-500	90.0%	89.6%	+0.4
GPQA finish=stop(收口)	198/198	186/198
GPQA finish=length(撞 32K 墙)	0	12

硬推理大涨(GPQA +7~13pp),知识不降,且蒸馏学会收口 —— 原版有 12 题撞 32K 墙(始终不出答案),蒸馏 198 题全收口。

MTP —— 原生多 token 预测(单流加速)

GGUF 内含原生 MTP 头(mainline llama.cpp --spec-type draft-mtp,无需外挂草稿模型)。DGX Spark GB10 单流实测,Q4_K_M:

--spec-draft-n-max	TPS	接受率	接受长度
裸(无 MTP)	10.4	—	—
n-max=3 ⭐	26.8	0.72	3.16

单流 2.3–2.6× 加速,无损(贪心投机只接受目标 argmax token)。投机是单流手段 —— 并发会摊薄它,高吞吐场景用裸多并发。

运行

llama-server -m Qwen3.6-27B-DSV4Pro-Distill-MTP-Q4_K_M-imatrix.gguf \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  -c 8192 --jinja --host 127.0.0.1 --port 8080

思考型模型 —— 必须 temp 0.6 / top_p 0.95(贪心会死循环)。图像输入加 --mmproj Qwen3.6-27B-DSV4Pro-Distill-mmproj-F16.gguf。

Claude Code（实验性 / experimental）

Claude Code 需要兼容 Anthropic /v1/messages 的后端 + 可靠的工具调用，不能直接喂 GGUF 文件路径。GGUF 走 LM Studio：LM Studio 0.4.1+ 加载本仓 GGUF（Q5_K_M 推荐），自带 Claude Code /v1/messages 端点，Claude Code 指过去即可。

vLLM 路线需 safetensors —— 用 FP8 仓 或 BF16 主仓，加 --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3，模型名用 --served-model-name（不带斜杠）。参考 LM Studio · vLLM Claude Code。

---

Claude Code (experimental)

Claude Code needs an Anthropic-compatible /v1/messages backend + reliable tool calling — you cannot hand it a GGUF path. GGUF goes via LM Studio: LM Studio 0.4.1+ loads this repo's GGUF (Q5_K_M recommended) and exposes a built-in Claude Code /v1/messages endpoint — just point Claude Code at it. The vLLM route needs safetensors — use the FP8 repo or the BF16 parent with --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3, model name = --served-model-name (no slashes).