stuchapin/Carnice-V2-27B-MTP-GGUF

Carnice-V2-27B + MTP — GGUF for Ampere/older NVIDIA

GGUF builds of kai-os/Carnice-V2-27b with the MTP (Multi-Token Prediction) speculative-decoding head grafted in, packaged for llama.cpp. The MTP tensors come from sakamakismile/Carnice-V2-27b-NVFP4-TEXT-MTP (whose NVFP4 build is Blackwell-only). This repo makes the same MTP-accelerated decoding available on Ampere / Ada / pre-Blackwell GPUs via GGUF.

---

⚠️ Requires unmerged llama.cpp PR #22673

These GGUFs only work with llama.cpp PR #22673, which adds MTP support and is still in draft as of 2026-05-11. Mainline llama.cpp will fail to load them.

git clone https://github.com/ggml-org/llama.cpp.git && cd llama.cpp
git fetch origin pull/22673/head:mtp-pr && git checkout mtp-pr
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=86
cmake --build build --target llama-server llama-quantize -j8

(CUDA_ARCHITECTURES=86 for RTX 3090; 89 for 4090/Ada, 90 for Hopper.)

---

Variants

File	Base quant	Size	Quality	Best for
Carnice-V2-27B-Q8_0-mtp.gguf	Q8_0	28 GB	A+ (near-lossless)	Max quality, 32GB+ VRAM or multi-GPU
Carnice-V2-27B-Q6_K-mtp.gguf	Q6_K	22 GB	A+	Best mainstream quality
Carnice-V2-27B-Q5_K_M-mtp.gguf	Q5_K_M	19 GB	A	Single 24GB card, A-tier quality
Carnice-V2-27B-Q4_K_M-mtp.gguf	Q4_K_M	17 GB	B+	Speed/quality balance
Carnice-V2-27B-IQ4_XS-mtp.gguf	IQ4_XS	15 GB	B+	Speed leader on Ampere

All variants keep the MTP block (blk.64.*) at BF16. Quantizing the MTP head didn't help speed and risked acceptance-rate degradation in testing.

---

Benchmarks — real agent workload

Measured on 2× RTX 3090 (Q8_0 and Q6_K via tensor-split, others single-GPU), --spec-draft-n-max 1, -c 32768, KV q4_0, flash attention on, thinking mode enabled. Tested with 7 representative prompts from a Hermes-style agent workload:

1. tool-call-meeting — emit JSON tool calls for a CRM workflow
2. morning-brief-synthesis — 5-bullet summary from agent context
3. concierge-permit-extract — structured extraction from permit text
4. draft-touch-card — short-form professional writing
5. lease-clause-explain — domain knowledge / explanation
6. long-context-summarize — 30-lead CRM dump → top 3
7. reasoning-lease-vs-buy — multi-step numerical reasoning

Aggregate

Variant	Mean tok/s	MTP acceptance	Total tokens / 7 prompts
Q8_0	34.0	82%	12,144
Q6_K	38.4	84%	12,680
Q5_K_M	39.9	83%	12,660
Q4_K_M	44.0	81%	12,084
IQ4_XS	47.7	82%	13,111

Per-prompt tok/s

Prompt	Q8_0	Q6_K	Q5_K_M	Q4_K_M	IQ4_XS
tool-call-meeting	28.7	26.9	32.5	38.3	35.0
morning-brief-synthesis	34.9	39.6	42.9	46.8	49.4
concierge-permit-extract	33.8	38.4	40.0	44.8	47.6
draft-touch-card	34.3	38.2	41.2	45.2	48.2
lease-clause-explain	33.7	38.9	39.9	44.6	48.7
long-context-summarize	34.7	38.9	40.0	44.6	48.0
reasoning-lease-vs-buy	33.2	37.6	38.7	41.9	46.6

MTP acceptance is steady 81–84% across all variants — quant choice didn't degrade the draft head's accuracy. Speed differences come almost entirely from the base model's memory bandwidth per token.

Reference: non-MTP vanilla decode on the same hardware sits at ~35 tok/s, so even the slowest MTP variant (Q8_0) is roughly even with vanilla; IQ4_XS gives ~1.4× over the vanilla baseline on this same workload mix.

---

Method

The whole thing is reproducible end-to-end. Headline steps:

1. Source materials

• Base weights: kai-os/Carnice-V2-27b (BF16 safetensors, 55 GB) — the Hermes-style SFT of Qwen3.6-27B.
• MTP head: 15 mtp.* BF16 tensors (~850 MB) extracted from sakamakismile/Carnice-V2-27b-NVFP4-TEXT-MTP. sakamakismile kept these tensors unquantized inside their NVFP4 file (the surrounding model is FP8/U8), so no dequantization required — just lift them out with safetensors.safe_open.

2. Merge

A small Python script symlinks the base BF16 shards into a working directory, writes the 15 MTP tensors as a new model-mtp.safetensors, regenerates model.safetensors.index.json, and rewrites config.json to set language_model_only: true (Carnice's base config carries a vision tower we don't ship).

3. Convert to GGUF

convert_hf_to_gguf.py from llama.cpp PR #22673 has first-class support for Qwen3_5ForConditionalGeneration and automatically remaps the mtp.* tensors to llama.cpp's blk.64.nextn.* naming. Output is a single BF16 GGUF (~55 GB).

4. Quantize

llama-quantize from the same PR build, with --tensor-type "blk\.64\..*=bf16" to keep the MTP block at full precision. Five targets: Q8_0, Q6_K, Q5_K_M, Q4_K_M, IQ4_XS.

5. The critical tuning find — --spec-draft-n-max 1

Default community guides recommend --spec-draft-n-max 3. For Carnice this is wrong by a wide margin. The MTP head has mtp_num_hidden_layers: 1 — it was trained to predict exactly one token ahead. Drafting more tokens recurses the same head autoregressively; errors cascade rapidly.

We swept --spec-draft-n-max values 1, 2, 3, 5, 7 on the same workload:

--spec-draft-n-max	Mean tok/s	Notes
1	52.6	Optimal — acceptance stays 80%+
2	(crash on our hardware)
3	46.6	Acceptance drops to 26% on creative outputs
5	38.5	Marginal acceptance, draft cost dominates
7	(crashed)

General rule: set --spec-draft-n-max = mtp_num_hidden_layers from the model's config.

6. What didn't help

We also exhaustively swept (each at --spec-draft-n-max 1):

• MTP block quant (BF16 / Q8_0 / Q5_K) — all within ~2% noise. Default to BF16 for safety.
• KV cache type (q4_0 / q8_0) — within noise; f16 OOMs at 64K context.
• Batch / ubatch sizes — within noise.
• Thread counts (1, 4, 8, auto) — within noise (the workload is GPU-bound).
• CUDA graph disable — flag absent in PR build.
• ik_llama.cpp fork — ~40% slower for this architecture. Carnice has 48 linear-attention (Mamba2-style) layers; mainline+PR-22673's kernels for those are noticeably more optimized than ik_llama.cpp's. Stick with mainline+PR.

---

Running

Single-GPU (≤19 GB models)

./build/bin/llama-server \
  -m Carnice-V2-27B-IQ4_XS-mtp.gguf \
  --alias carnice-v2-mtp \
  -c 65536 --parallel 1 -ngl 99 \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  --flash-attn on --no-context-shift \
  --jinja --reasoning-format deepseek --reasoning-budget 4096 \
  --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 \
  --batch-size 2048 --ubatch-size 1024 --metrics \
  --spec-type mtp --spec-draft-n-max 1 \
  --fit off \
  --host 127.0.0.1 --port 8001

Dual-GPU (Q6_K / Q8_0)

Add --tensor-split 14,14 and CUDA_VISIBLE_DEVICES=0,1. Expect ~15% PCIe overhead vs a single ≥32GB card.

Verify MTP is active

In the server startup log:

set_mtp: MTP draft head registered (ctx_mtp=..., n_embd=5120)
slot   load_model: speculative decoding context initialized

In per-request response timings:

"timings": {
  "predicted_per_second": 47.6,
  "draft_n": 13,
  "draft_n_accepted": 13
}

draft_n_accepted / draft_n should be ≥ 70% — we routinely see 100% on tool-call workloads.

---

Build recipe (reproduce from source)

# Merge MTP tensors into BF16 base
import json
from pathlib import Path
from safetensors import safe_open
from safetensors.torch import save_file

BASE = Path("./kai-os--Carnice-V2-27b")
MTP_SRC = Path("./sakamakismile--Carnice-V2-27b-NVFP4-TEXT-MTP")
OUT = Path("./merged")
OUT.mkdir(exist_ok=True)

for f in BASE.iterdir():
    if f.name in ("model.safetensors.index.json", "config.json"): continue
    (OUT / f.name).symlink_to(f.resolve())

mtp = {}
with safe_open(MTP_SRC / "model.safetensors", framework="pt") as fp:
    for k in fp.keys():
        if k.startswith("mtp."):
            mtp[k] = fp.get_tensor(k)
save_file(mtp, str(OUT / "model-mtp.safetensors"))

with open(BASE / "model.safetensors.index.json") as f: idx = json.load(f)
for k in mtp: idx["weight_map"][k] = "model-mtp.safetensors"
with open(OUT / "model.safetensors.index.json", "w") as f: json.dump(idx, f)

with open(BASE / "config.json") as f: cfg = json.load(f)
cfg["language_model_only"] = True
for k in ("vision_config", "image_token_id", "video_token_id"):
    cfg.pop(k, None)
with open(OUT / "config.json", "w") as f: json.dump(cfg, f, indent=2)

# Convert merged HF checkpoint to BF16 GGUF
python convert_hf_to_gguf.py ./merged \
    --outfile Carnice-V2-27B-bf16-mtp.gguf --outtype bf16

# Quantize, preserving the MTP block at BF16
./build/bin/llama-quantize --tensor-type "blk\.64\..*=bf16" \
    Carnice-V2-27B-bf16-mtp.gguf Carnice-V2-27B-IQ4_XS-mtp.gguf IQ4_XS

---

Known issues

1. PR #22673 trips free(): invalid pointer SIGABRT on graceful shutdown. Runtime is clean; the kernel reclaims the process. Cosmetic. Likely fixed before PR merges.
2. Vision input + MTP crashes in PR #22673. These GGUFs are text-only (vision config stripped).
3. Prefill is ~half-speed with MTP enabled vs non-MTP. Matters most for very long prompts; decode is where MTP wins back.

---

License & attribution

This work is Apache 2.0 (inherits from kai-os/Carnice-V2-27b). The underlying Qwen3.6 base is subject to the Tongyi Qianwen License — commercial use exceeding 700K MAU requires a separate license from Alibaba.

Credit chain

• Qwen team (Alibaba) — Qwen3.6 base model
• kai-os — Carnice-V2-27B SFT (Hermes-style agent training)
• sakamakismile — MTP head graft + NVFP4 packaging; we extracted the MTP tensors from their build
• am17an and llama.cpp contributors — MTP support via PR #22673
• stuchapin (this repo) — GGUF + MTP fusion for Ampere, --spec-draft-n-max 1 tuning finding, end-to-end agent-workload benchmarks

---

Changelog

• 2026-05-11 — Initial release. 5 GGUF variants (Q8_0, Q6_K, Q5_K_M, Q4_K_M, IQ4_XS), all clean from BF16 source, MTP block preserved at BF16. llama.cpp PR #22673 commit 5d5f1b4. Real-workload benchmarks across 7 Hermes-style agent prompts.