Kirigami: Zero-Shot Expert Carves of Qwen3.6
Collection
Qwen3.6-35B-A3B carved by expert importance to fit one consumer GPU. No training, no calibration. 800 tok/s on a laptop 5090. • 4 items • Updated
Kirigami-Qwen3.6-28B-A3B-NVFP4
Zero-shot expert carve of Qwen3.6-35B-A3B — no training, no calibration. 256 → 192 experts per layer (25% dropped), carved directly from the NVFP4 checkpoint by importance, specifically so the model fits and runs fast on a single 24GB GPU (RTX 5090 laptop) where the full 35B cannot allocate KV.
The Kirigami family
Zero-shot expert carves of Qwen3.6-35B-A3B, sized for single consumer GPUs (kirigami: the Japanese art of cutting paper to create structure — here, cutting experts to fit the card). No training, no calibration, one CPU pass.
| Model | Experts kept | Params | Weights | Boots on 24GB at |
|---|---|---|---|---|
| Kirigami-Qwen3.6-28B-A3B-NVFP4 | 192/256 | 27.9B | 19.8 GiB | 8K ctx |
| Kirigami-Qwen3.6-24B-A3B-NVFP4 | 160/256 | 23.9B | 17.3 GiB | 16K ctx |
| Kirigami-Qwen3.6-20B-A3B-NVFP4 | 128/256 | 19.8B | 14.9 GiB | 32K-class ctx |
Method
One-shot, shard-streamed CPU pass over unsloth/Qwen3.6-35B-A3B-NVFP4 —
top-K expert selection (keep/drop, no merging), described fully below so the
method is reimplementable:
- Importance (checkpoint-resident, zero forward passes):
I_e = ||router_row_e||₂ × mean(weight_scale_e)— router propensity × NVFP4 scale energy, per layer. - Kept experts renumbered contiguously; router rows sliced in the same order (routing semantics preserved). Shared expert and all attention untouched.
carve_manifest.json(shipped in this repo) records the kept-expert indices per layer.
26 GB → 19.8 GiB (46 MoE layers × 192 experts). Carve time: ~40 min CPU.
Measured on RTX 5090 laptop (24GB), vLLM v0.24, 4.4K-token prompt
| Mode | TTFT | Decode | 4-way aggregate |
|---|---|---|---|
| enforce-eager | 394 ms | 16 tok/s | 29 tok/s |
| compiled (CUDA graphs) | 293 ms | 169 tok/s | 174 tok/s |
The FULL 35B fails KV allocation on this card in every tested config (deficit −0.23 GiB at 8K ctx); this carve boots with headroom at 8K.
Measured throughput (RTX 5090 laptop 24GB, vLLM compiled, v2 carve)
Long-prompt workloads:
| Workload | Single-stream | Concurrent |
|---|---|---|
| 4.4K prompt, 1024-tok gen | 168 tok/s (TTFT 288 ms) | 574 @8-way / 805 @16-way |
| 5.5K (near-max) prompt, 2048-tok gen | 167 tok/s (TTFT 345 ms) | 391 @4-way |
Long-decode workloads (ignore_eos, thinking-style):
| Workload | Single-stream | Concurrent |
|---|---|---|
| 300-tok prompt, 4096-tok decode | 169 tok/s (TTFT 70 ms) | 431 tok/s @4-way |
| 300-tok prompt, 7.5K-tok decode | 168 tok/s | 260 tok/s @2-way |
Sustained throughput under real concurrent load (12 in-flight requests, long multi-thousand-token prompts — the most representative numbers):
| Metric | Value |
|---|---|
| Peak prefill | 3,319 tok/s |
| Peak generation (aggregate) | 746 tok/s |
| Sustained generation (aggregate) | 537 tok/s |
| Peak KV-cache utilization | ~31% |
| Concurrent requests | 12 |
Decode speed is nearly flat in both prompt depth and generation length (hybrid linear-attention backbone); prefill ingests ~11K tok/s. Includes a carved MTP draft head (fused tensors sliced consistently) for speculative decoding.
MTP speculative decoding (included)
The parent's MTP draft head is carved consistently (fused draft-expert
tensors + draft router sliced with their own importance set) and works with
vLLM's qwen3_5_mtp speculative method. Measured on the 20B rung
(temp 0, log-analysis prompts — acceptance is workload-dependent):
| Workload | No MTP | num_speculative_tokens: 4 |
|---|---|---|
| 1024-tok gen, single | 170 tok/s | 239 tok/s (+41%) |
| 4096-tok decode, single | 169 tok/s | 259 tok/s (+54%) |
| 4096-tok decode, 4-way | 438 tok/s | 657 tok/s (+50%) |
--speculative-config '{"method": "qwen3_5_mtp", "num_speculative_tokens": 4}'
Serve
vllm serve <this-repo> --kv-cache-dtype fp8 --max-model-len 8192 \
--gpu-memory-utilization 0.94 --max-num-seqs 2
Notes: NVFP4 MoE requires the cutlass/flashinfer backends (do NOT set
moe_backend=triton). Thinking-mode preamble is on by default — disable via
chat_template_kwargs: {"enable_thinking": false} if you want direct answers.
Honest caveats
- Lossy by construction: 25% of experts are gone. Coherence smokes (log RCA, constraint following, factual explanations) show no visible degradation; no benchmark suite has been run. Certify before production.
- Importance is uncalibrated (weight/scale statistics only). A calibration-based selection (router activation statistics) would likely select better; this artifact demonstrates the zero-cost floor.
- Vision tower weights are carried over but untested post-carve.
- Downloads last month
- 22