zhangrover/nerkyor-Qwen3.8-27B-EfficientThink-KQVQ4-IQ3Floor-10G-MTP-Fast-GGUF

Qwen3.8-27B EfficientThink · SimPO · KQV-Q4 + IQ3 Floor · MTP Fast

English ｜ 中文（点击跳转）

My previous version, nerkyor-Qwen3.8-27B-EfficientThink-SimPO-KQVQ4-IQ3Floor-10G-MTP-GGUF, has had more than 1,500 downloads and at least 3 likes as of 2026-09-23. It is a single-file build with a working built-in MTP head, friendly to machines that sit under 16 GB of VRAM. But can it be even faster?

Yes. This build replaces the draft model. The new MTP head comes from Jackrong/Qwopus3.8-27B-Flash-GGUF: attention runs at Q4_K, attn_v / ffn_down sit at full Q6_K, so more drafts get accepted per step than before. On the same RTX 5060 Ti (16 GB) at ~130k context, where the previous build peaked around 40 to 50 tok/s, this one occasionally goes past ~50. Cards with a lower base decode speed still pick up +5 tok/s or more from the new head alone. If you keep just one of these builds, keep this one. Thanks to Jackrong for publishing the Flash build in a form that makes this head possible.

Nothing moved outside that head: all 851 non-head tensors keep their exact allocation from the previous build (the complete model stays at 866 = 851 trunk + 15 MTP head; block_count=65, nextn_predict_layers=1). The speed numbers above come from stock llama.cpp, b10955 with CUDA 13.3. Community-maintained forks sometimes ship optimizations upstream does not have yet; if one of those is what you run, there may be room left to find on your own.

Built on Qwen3.8; thanks to the Qwen team, and to nerkyor for the base this is quantized from.

Files

File	Size	Role
Qwen3.8-27B-EfficientThink-SimPO-KQVQ4-IQ3Floor-10G-MTP-Fast.gguf	10,948,191,968 B (10.19 GiB)	KQV-Q4 + IQ3 Floor build with the new MTP head
mmproj-Qwen3.8-27B-Q8_0.gguf	629,247,008 B (0.60 GiB)	Qwen3.8 vision projector, paired via --mmproj and optional

SHA1: the main model is 9cc69bf32f915674efe7ab714c95f725575993a9384bed32e3d15fc6e66d9e11; the vision projector is 892a57fac34c1685ce2b4b09e085f809a0a13b64.

Changes versus the previous build

Only precision moved, across a total of +76.3 MiB:

Role	Previous build	Fast
blk.64 attn_q / attn_k / attn_output, ffn_gate / ffn_up, nextn.eh_proj	IQ3_S	Q4_K
blk.64 attn_v, ffn_down	Q4_K (attn_v), IQ3_S (ffn_down)	Q6_K both

Type histogram: Q4_K 49 · IQ3_S 130 · IQ3_XXS 327 · F32 360 became Q4_K 54 · IQ3_S 123 · IQ3_XXS 327 · Q6_K 2 · F32 360. Everything outside blk.64 keeps the exact allocation of the previous build.

Usage

Tested on llama.cpp b10955 with CUDA 13.3, where drafts are accepted live (draft_n and draft_n_accepted both above zero):

llama-server.exe `
  -m Qwen3.8-27B-EfficientThink-SimPO-KQVQ4-IQ3Floor-10G-MTP-Fast.gguf `
  --mmproj mmproj-Qwen3.8-27B-Q8_0.gguf --image-min-tokens 1024 `
  -c 260000 --kv-unified --cache-type-k q4_0 --cache-type-v q4_0 `
  -ngl all --fit off -t 8 -b 1024 -ub 128 -n 65536 `
  --temp 1.0 --top-k 20 --top-p 0.95 --repeat-penalty 1.0 --min-p 0.0 `
  --spec-type draft-mtp -np 1 --spec-draft-n-max 3 --spec-draft-p-min 0 -ngld 99 `
  -fa on --mlock --cont-batching --no-mmap --no-warmup `
  -ctxcp 64 --cache-ram 2048 `
  --jinja --chat-template-file "chat_template v22.3.jinja" `
  --reasoning-format deepseek --reasoning on --reasoning-budget 8192 `
  --reasoning-budget-message "... reasoning budget exceeded. Let's answer now." `
  --reasoning-preserve --slot-save-path kvcache

• --spec-type draft-mtp turns on MTP speculative decoding; drafts are accepted live. The draft head is small, so running it from CPU costs almost no speed.
• Same for the Q8_0 vision projector that ships in this repo (--mmproj line): about 0.6 GiB stays out of VRAM when it runs on CPU, and whatever that frees goes into KV cache.
• Pick -c at ~130k with a 16 GB card plus q4_0 unified KV; if your VRAM is larger than that, raise the number (the command above caps it at 260000).
• Reasoning runs out of the box: --reasoning-format deepseek with an 8192-token budget, and --reasoning-preserve keeps the thinking chain.
• In LM Studio you import the main GGUF and enable speculative decoding (MTP); no command line needed.

Verification

Check	Result
Main model tensors	866 = 851 trunk + 15 MTP head (block_count=65, nextn_predict_layers=1)
Trunk versus the previous build	byte-identical allocation; only blk.64 precision moved
Live test on b10955 at port 8081	clean start with MTP enabled; /completion returned draft_n=51, draft_n_accepted=25

Disclaimer

This model is provided AS IS without warranty of any kind; the Apache 2.0 license terms apply. It is a personal research build intended for individual research and evaluation, not public or production deployment. As an uncensored mixed-precision build its outputs are unpredictable, so you are responsible for your own prompts and outputs. Test in your own environment before relying on anything it generates; all risk is yours, and the author assumes no liability.

Base model: nerkyor/Qwen3.8-27B-EfficientThink-Uncensored-K3-Opus5-Grok4.6-GPT5.6Sol-SFT-SimPO-DFlash2. MTP head source: Jackrong/Qwopus3.8-27B-Flash-GGUF, with thanks to Jackrong.

---

中文说明

我的上一版本 nerkyor-Qwen3.8-27B-EfficientThink-SimPO-KQVQ4-IQ3Floor-10G-MTP-GGUF，截至 2026.9.23，下载量超过 1500、点赞至少 3；单文件构建、内置 MTP 头能跑，对低于 16 GB 显存的设备比较友好。但是还能更快吗？

能。这个版本的答案是把草稿模型整个换掉。新的 MTP 头取自 Jackrong/Qwopus3.8-27B-Flash-GGUF：注意力权重在 Q4_K，attn_v / ffn_down 满格 Q6_K，每一步被接受的草稿比过去更多。同一张 RTX 5060 Ti（16 GB）、约 13 万上下文条件下，上一版峰值在 40~50 tokens/s；这一版偶尔能超过 ~50，基础解码速度不高的显卡也能靠新头多出 +5 tokens/s 以上。如果只留一个版本，就留这个。感谢 Jackrong 发布 Flash 构建，才让这个头成为可能。

换掉头部之外什么都没动：851 个非头部张量全部沿用上一版的精确分配（完整模型仍是 866 = 851 主干 + 15 MTP 头；block_count=65、nextn_predict_layers=1）。上面的速度数字全部来自标准版 llama.cpp（b10955、CUDA 13.3）的实测。社区维护的魔改分支有时带着上游还没有的优化；如果你跑的是其中一种，可能还有余地去挖一挖。

基于 Qwen3.8；感谢 Qwen 团队，也感谢底模作者 nerkyor。

文件

文件	大小	角色
Qwen3.8-27B-EfficientThink-SimPO-KQVQ4-IQ3Floor-10G-MTP-Fast.gguf	10,948,191,968 B（10.19 GiB）	KQV-Q4 + IQ3 Floor 构建，带新 MTP 头
mmproj-Qwen3.8-27B-Q8_0.gguf	629,247,008 B（0.60 GiB）	Qwen3.8 视觉投影器，经 --mmproj 配对，可选

SHA1：主模型为 9cc69bf32f915674efe7ab714c95f725575993a9384bed32e3d15fc6e66d9e11，视觉投影器为 892a57fac34c1685ce2b4b09e085f809a0a13b64。

相对上一版的改动

只有精度在动，总体积 +76.3 MiB：

角色	上一版	Fast
blk.64 attn_q / attn_k / attn_output、ffn_gate / ffn_up、nextn.eh_proj	IQ3_S	Q4_K
blk.64 attn_v、ffn_down	Q4_K（attn_v）、IQ3_S（ffn_down）	均为 Q6_K

类型直方图：由 Q4_K 49 · IQ3_S 130 · IQ3_XXS 327 · F32 360 变为 Q4_K 54 · IQ3_S 123 · IQ3_XXS 327 · Q6_K 2 · F32 360。blk.64 之外的张量沿用上一版的分配。

用法

llama.cpp b10955 + CUDA 13.3 实测，草稿实机被接受（draft_n、draft_n_accepted 均大于零）：

llama-server.exe `
  -m Qwen3.8-27B-EfficientThink-SimPO-KQVQ4-IQ3Floor-10G-MTP-Fast.gguf `
  --mmproj mmproj-Qwen3.8-27B-Q8_0.gguf --image-min-tokens 1024 `
  -c 260000 --kv-unified --cache-type-k q4_0 --cache-type-v q4_0 `
  -ngl all --fit off -t 8 -b 1024 -ub 128 -n 65536 `
  --temp 1.0 --top-k 20 --top-p 0.95 --repeat-penalty 1.0 --min-p 0.0 `
  --spec-type draft-mtp -np 1 --spec-draft-n-max 3 --spec-draft-p-min 0 -ngld 99 `
  -fa on --mlock --cont-batching --no-mmap --no-warmup `
  -ctxcp 64 --cache-ram 2048 `
  --jinja --chat-template-file "chat_template v22.3.jinja" `
  --reasoning-format deepseek --reasoning on --reasoning-budget 8192 `
  --reasoning-budget-message "... reasoning budget exceeded. Let's answer now." `
  --reasoning-preserve --slot-save-path kvcache

• --spec-type draft-mtp 打开 MTP 投机解码，草稿实机被接受。MTP 头很小，放在 CPU 上跑对速度几乎没有影响。
• 本仓库自带的视觉投影器（--mmproj）同理：约 0.6 GiB 的 Q8_0 头留在 CPU 上，腾出的显存全部进 KV 缓存。
• -c 选 ~13 万即可在 16 GB + q4_0 统一 KV 下实测跑通；显存更大的话可以把数字往上加（上面命令里用的是 260000）。
• 思考开箱即用：--reasoning-format deepseek、预算 8192 token，--reasoning-preserve 保留思考链。
• LM Studio 里导入主 GGUF，服务端设置开启投机解码（MTP）即可，不需要命令行。

验证

项目	结果
主模型张量数	866 = 851 主干 + 15 MTP 头（block_count=65、nextn_predict_layers=1）
主干对照上一版	分配逐字节一致；只有 blk.64 精度提升
实机测试（b10955，端口 8081）	MTP 开启下干净启动；/completion 返回 draft_n=51、draft_n_accepted=25

免责声明

本模型按原样（AS IS）提供，不作任何明示或暗示的保证；以 Apache 2.0 许可证条款为准。这是个人研究构建，供个人研究与评估之用，不适合公共或生产部署。作为无审核（uncensored）混合精度构建，输出不可预期：使用者对自己的提示与输出负责。请在自己的环境中充分测试后再决定是否依赖其生成内容；风险自负，作者不对任何损失承担责任。

底模：nerkyor/Qwen3.8-27B-EfficientThink-Uncensored-K3-Opus5-Grok4.6-GPT5.6Sol-SFT-SimPO-DFlash2。MTP 头来源：Jackrong/Qwopus3.8-27B-Flash-GGUF，致谢 Jackrong。