isneezekittens/Carwin-MoE-Nano-GGUF

Carwin-MoE-Nano (GGUF)

A tiny, fast, local MoE model: a DARE-TIES merge of reasoning-heavy Darwin and agent/tool-calling-heavy Carnice, on the Qwen3.6-35B-A3B Mixture-of-Experts base, with the Qwen3.6 MTP (multi-token prediction) head preserved for self-speculative decoding — then quantized down to ~2-bit (IQ2, 2.78 bpw) with mudler's APEX recipe. It sits at 11 GB on disk and runs on modest hardware while keeping the reasoning and tool-calling of much larger models.

Why this exists

Built by a tech/AI hobbyist running a local agent stack on a 32GB Mac Studio. The goal was a model that fills the gap between small 9B models (not capable enough) and full 27B–35B Q4 quants (too big or too slow): a fast, capable MoE that fits comfortably and stays usable for real agent work — reasoning and tool-calling — at a fraction of the size.

What it is

Base	Qwen/Qwen3.6-35B-A3B (MoE, 256 experts, ~3B active)
Reasoning parent	FINAL-Bench/Darwin-36B-Opus
Agent / tool-calling parent	samuelcardillo/Carnice-Qwen3.6-MoE-35B-A3B
Merge method	DARE-TIES (50/50, density 0.53 each, BF16 master)
Format	GGUF (llama.cpp)
Quantization	~2-bit mixed (APEX recipe): IQ2_XXS mid experts, higher-precision edges/attention/shared-expert; MTP head pinned Q8_0
Bits per weight	2.78 bpw
Size on disk	~11 GB
MTP	19-tensor MTP head grafted from Carnice, preserved at Q8_0/F32 (not crushed to 2-bit)
Context	up to 262K (model trained window); served at any cap you choose
License	Apache-2.0 (all three parent lines permissive)

How it was built

The entire pipeline — merge, MTP graft, quantization, and verification — was orchestrated by a local Hermes agent (Nous Research) running on a single 32GB Mac Studio (M2 Max). No cloud GPUs, no rented compute; every step ran on-device, agent-driven from start to finish.

1. Merge — DARE-TIES merge of Darwin (reasoning) and Carnice (tool-calling) against the Qwen3.6-35B-A3B MoE base, producing a full-precision BF16 master. Architecture compatibility was verified across all three (same 40 layers, 256 experts, identical vocab) before merging.
2. MTP graft — the base/Darwin side lacked an MTP head, so the 19-tensor MTP head was grafted from Carnice (which carries it) into the merged master, written as a separate shard. Every MTP tensor was byte-verified against the source — not just trusted from the index — since silent MTP drop is the known failure mode for this kind of work.
3. Validation — the BF16 master was confirmed to reason correctly and emit clean tool calls before any quantization, so the merge itself was known-good first.
4. APEX quantization — quantized to ~2-bit using mudler's APEX recipe: a MoE-aware mixed-precision profile (IQ2_XXS on mid-layer routed experts, higher precision on edge layers, attention, and the shared expert), guided by an importance matrix (imatrix). The MTP/nextn tensors were force-pinned to Q8_0 so the draft head stays near-lossless.
5. Byte verification — the final Nano was dumped and checked: MTP tensors present and at Q8_0 (not IQ2), the nextn_predict_layers metadata key intact.

Running (llama.cpp)

Requires a recent llama.cpp build with Qwen3.6 MTP support. Serve with MTP speculative decoding enabled:

llama-server -m carwin-moe-Nano.gguf --spec-type draft-mtp --spec-draft-n-max 1 -ngl 99 -c 8192

Notes:

• It's a MoE (~3B active of ~35B total), so generation is fast for its capability class — that's the point of the size.
• The MTP head enables self-speculative decoding for extra generation speed; whether it's engaged is the --spec-type draft-mtp flag.
• llama.cpp ships a built-in web UI — open the server's port in a browser to chat with it directly, MTP and all.

KV cache: keep it at full precision

Run with the KV cache unquantized (FP16). Because the weights are already at ~2-bit, quantizing the KV cache on top of that compounds the precision loss and the model starts producing unreliable / incorrect output ("lying"). The compression budget is already spent on the weights — leave the cache full-precision.

In llama.cpp this means not passing --cache-type-k / --cache-type-v quantization flags (the default FP16 cache is what you want).

Note: weight + KV-cache quantization only coexist cleanly with methods that quantize both jointly (e.g. TQ, Prism). This model uses APEX (weights only), so KV-cache quantization should stay off — see above.

Evaluation

Measured locally with lm-evaluation-harness on 200-sample subsets (--limit 200) of each benchmark — indicative quick-checks, not full leaderboard runs. Thinking was handled correctly (scores reflect the final answer, not reasoning traces):

Benchmark	Score (200-sample)	Notes
GSM8K (math reasoning)	~75.5%	Strong — the reasoning lineage survived 2-bit
IFEval (instruction-following)	~27% prompt-level / ~42% instruction-level	Weaker — this is a reasoning/tool model, not a rigid-format model
MTP draft acceptance	~85% / ~76%	Excellent — the grafted head matches the merged body well (from smoke tests)

These are quick subset evaluations, not exhaustive benchmark runs — treat them as a rough profile, not precise rankings. The honest takeaway: strong reasoning, capable tool-calling, weak at rigid instruction-formatting — a coherent trade for an agent/reasoning model at this size and bit-rate. Some of the IFEval weakness is the 2-bit quantization tax; a higher-precision build of the same merge would likely score higher there.

Performance (tokens/sec) depends heavily on hardware and isn't quoted here — measure on your own machine.

Known quirks

• Identity: the model may misidentify itself (e.g. as Gemini, or guess the wrong base/runtime). This is cosmetic lineage residue, not a fault — it has no reliable knowledge of its own construction.
• 2-bit edge cases: on adversarially-phrased reasoning ("all but 9 die" style traps), it can misparse where a higher-precision build wouldn't. Core reasoning on clean problems is strong.
• Format compliance: won't reliably nail rigid output constraints (exact word counts, strict templates) — see IFEval above.
• MTP preserved but acceptance is merge-dependent: the head is physically in the file and accepts well here (~85%), but acceptance on a merged body can differ from stock; if low on your setup, run without --spec-type draft-mtp for identical-quality output.

Credits

All credit to the authors of the parent models and base: FINAL-Bench/Darwin-36B-Opus, samuelcardillo/Carnice-Qwen3.6-MoE-35B-A3B, and Qwen/Qwen3.6-35B-A3B. Merged with mergekit; quantized with mudler's APEX recipe.