--- license: other license_name: qwen license_link: https://huggingface.co/Qwen/Qwen3.6-27B/blob/main/LICENSE language: - en tags: - merge - task-vector - dare-ties - layer-weighted - qwen3 - qwen35 - reasoning - code - gguf - bf16 - full-precision - llama.cpp - experimental base_model: - Qwen/Qwen3.6-27B pipeline_tag: text-generation library_name: gguf ---
Qwopus3.6-27B Fusion — one 27B that reasons and codes, no skill lost

▶ PLAY ORBITAL — a game this model built, live in your browser ◀

> **⬛ Full-precision BF16 weights.** This repo hosts the **unquantized BF16 GGUF** (~54.7 GB) — the > reference weights every quant is derived from. **Want a smaller file that runs on one GPU?** The > quantized GGUFs (`Q3_K_M`, `Q4_K_M`, `Q5_K_M`, `Q6_K`) live in the companion repo → > **[`KyleHessling1/Qwopus3.6-27B-Fusion-GGUF`](https://huggingface.co/KyleHessling1/Qwopus3.6-27B-Fusion-GGUF)**. > **Research preview.** This model attempts to combine the **reasoning** capability of > `Qwopus3.6-27B-v2` with the **code-execution** capability of `Qwopus3.6-27B-Coder` — in a single 27B > model, *without significant loss to either* skill. In practice it behaves like a production model > with both capabilities fused, but it has **not** yet been through a full rigorous evaluation. If you > find issues, please reach out on X — **[@kylehessling1](https://x.com/kylehessling1)**. - **Format:** GGUF, **`BF16`** full precision (~54.7 GB) — the source weights; needs ~55 GB of RAM/VRAM (multi-GPU or CPU/`llama.cpp` offload). For single-GPU use, grab a quantized GGUF from the [companion repo](https://huggingface.co/KyleHessling1/Qwopus3.6-27B-Fusion-GGUF). - **Architecture:** `qwen35` hybrid (linear-attention + periodic full-attention), 27B params, 64 layers. - **Native context:** 262,144 tokens. Verified clean (needle + termination) to 60K. - **Modes:** thinking (reasoning) **on** by default; strong agentic/coding behavior. ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## ◆ TL;DR | | | |---|---| | **Base / anchor** | `Qwen/Qwen3.6-27B` (both parents descend from it) | | **Parent A — reasoning** | `Qwopus3.6-27B-v2` — donates the merge base + embeddings/head/norms/MTP | | **Parent B — code** | `Qwopus3.6-27B-Coder` — delta source, injected with depth-increasing weight | | **Merge math** | `W(L) = V2 + α(L)·(Coder − V2)`, α linear ramp **0.12 (early) → 0.48 (late)** | | **Frozen (copied from A)** | `embed_tokens`, `lm_head`, all `norm`, `mtp`/NextN, any vision tensors | **Headline results** (Q4_K_M, thinking-on unless noted): | Capability | Score | Stability | Score | |---|---|---|---| | HumanEval | **94.5%** | Termination @ temp 0.2 & 0.9 | **4/4 clean** | | MBPP | **87.9%** | Temp sweep 0.5–0.9 | **10/10 clean** | | GSM8K | **95.0%** | Seed variance (HumanEval) | **std 0.0** | | MATH-500 | **62–64.7%** | Long-context needle+term @ 60K | **perfect** | | IFEval (strict) | **82.8%** | SWE-bench Verified (astropy, 0:15) | **7/15 (47%)** | > Against the alternatives on the same astropy slice: **Fusion 7/15** > equal-weight DARE-TIES **5/15** > DUS-38B frankenmerge **3/15**. ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## ◆ Where it shines — the one-stop creative programming agent > **Best used as a single, closed-loop agentic creative programmer.** In a *build → run → self-fix* > loop, Fusion is at its strongest as **one agent that plans, writes, and debugs on its own** — the > reasoning⊕coding balance lets it close the loop by itself (see the ORBITAL / FLUX demos below, > where it root-caused and fixed its own bugs from Playwright error reports). > > This is a different niche than its parents. The **`Coder`** model is excellent for **coding under > active steering**, or as a **solid orchestration model** driving other tools/agents. **Fusion**, by > contrast, is the ideal **one-stop creative builder** — you hand it a goal and it carries the whole > creative-programming loop end-to-end. ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## ◆ Why I chose to make this merge Two finetunes of the same base diverge into complementary specialists: one gets better at multi-step reasoning, the other at writing code — but each pays for it. In particular the **coding finetune becomes unstable in thinking mode** (it loops / degrades at low temperature). The goal was a single model that keeps the reasoner's stability *and* the coder's skill. My naive approaches, one of them eroneously proposed by Claude Fable, average the two, or run an equal-weight DARE-TIES merge from the common base. These alternatives underperformed and the geometry explains why. ### ▌ The lineage Because the coder model descended from the v2 version, the coders additions can be evaluated against the v2 by analyzing the delta. Projecting the coding delta onto the reasoning delta reveals: ``` proj(ΔCoder · ΔV2 / |ΔV2|²) ≈ 0.4 in early layers → ≈ 1.0 in late layers ``` That ramp to 1.0 means **late layers of the coder already contain V2's full reasoning delta**. And: - The coding delta is **~1.8× the magnitude** of the reasoning delta. - In **early layers** the coding delta is **anti-aligned** with reasoning (`cos < 0`) — the coder *partially reverted* V2's early-layer reasoning/control representations. **That anti-alignment is the measured source of the coder's thinking-mode instability.** An equal-weight merge (both task vectors added from the base) therefore commits two errors: 1. It **double-counts V2's reasoning delta** (the coder already contains it), and 2. It **imports the full early-layer destabilizing coding delta** at 100% strength. ### ▌ The fix Because the coder sits "downstream" of V2, the correct anchor is **V2 itself**, not the shared base — this avoids the double-count. Then, instead of a constant fraction of the coding delta everywhere, inject it with a **depth-increasing schedule**: - **Early layers (α ≈ 0.12):** inject *little* coding delta → **protect** V2's reasoning/control reps exactly where the coder damaged them. - **Late layers (α ≈ 0.48):** inject *more* coding delta → add coding skill where the two parents already agree (proj → 1.0) and it's safe. The shape of the α ramp is dictated by the geometry, not hand-tuning. Embeddings, LM head, norms, and the MTP head are identical across all three models (delta ≈ 0), so they're copied verbatim from V2 rather than merged.
Weight geometry — coder descends from V2, its delta is 1.8x and anti-aligned in early layers; the alpha ramp injects coding 0.12 early to 0.48 late
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## ◆ How it was built **Merge.** The parents use Qwen's **hybrid attention schedule** (most layers linear/DeltaNet; every 4th layer full softmax attention). Off-the-shelf `mergekit` **cannot** merge this architecture — it templates `linear_attn.*` onto every layer and errors on the full-attention layers. So the merge is a **custom streaming task-vector routine** operating directly on the (identical) tensor keys, immune to arch templating. The full, reproducible script ships in this repo as [`merge_layerweighted.py`](./merge_layerweighted.py): ```python def alpha(L, a_early, a_late): return a_early + (a_late - a_early) * (L / (NLAYERS - 1)) # smooth linear ramp by depth # per decoder tensor at layer L: out = (v2 + alpha(L, 0.12, 0.48) * (coder - v2)).to(torch.bfloat16) # embeddings / lm_head / norms / mtp / vision: copied from V2 verbatim ``` **Quantize.** The bf16 merge was converted to GGUF and quantized to **`Q4_K_M`** with `llama.cpp` (851 GGUF tensors, 16.5 GB). **Evaluate.** Every candidate was gated at **both** temp 0.2 (a termination stress test) and temp 0.9 (the finetune operating point) — "clean only at high temp" is riding sampling entropy, not stability. This model is clean at both. ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## ◆ Demonstrations — driving a local agentic coding loop Every artifact below was generated by **this model** running as the backend of an agentic coding loop (OpenCode / Claude Code), building single-file HTML/JS apps end-to-end, then **verified with headless Playwright**. What stood out isn't just that it writes polished games — it's that when the Playwright harness handed back error reports, **the same model root-caused and fixed its own bugs** with precise, correct diagnoses (excerpts below). ### 🚀 ORBITAL — physics space-arcade (thinking-on) *Setup:* "Build a complete, physics-driven, visually stunning arcade space game as a single self-contained index.html — gravity is the weapon." *Outcome:* **built and runs cleanly, zero console errors**, real 2-body gravity, particle trails, wave HUD, and it even self-balance-tuned its own difficulty. 1,150 lines.

▶ PLAY IT LIVE → huggingface.co/spaces/KyleHessling1/ORBITAL ◀

(runs entirely in your browser — the exact single file this model wrote)
| Title screen | Live gameplay (Wave 1) | |---|---| | ![ORBITAL title](assets/orbital-title.png) | ![ORBITAL gameplay](assets/orbital-gameplay.png) | The same loop also produced three more complete, self-contained games — **PRISM DEFENSE** (neon tower defense, 1,074 lines), **NEON SERPENT** (synthwave Snake with smooth interpolated movement, 657 lines), and **STARLIGHT SURVIVORS** (survivors-style auto-shooter, 555 lines). Each was built, Playwright-tested, and where the harness surfaced a bug the model root-caused and fixed it itself (e.g. a tower-targeting `bestDist = Infinity` that should have been `-Infinity`; three weapon upgrades missing a `fireTimer` init that made `fireTimer -= dt` evaluate to `NaN`). The impressive part is the quality of the model's own debugging inside a *build → error report → self-fix* loop, not flawless one-shot output. ### 📋 FLUX — a full Trello-style Kanban dashboard (the strongest capability demo) The most demanding build: a complete **single-file Kanban app** (`~3,373 lines`, no backend, localStorage persistence), assembled by the model over **five steered phases**, each a fresh session that read a spec + the current file and extended it — Playwright-verified between phases. ![FLUX board](assets/flux-board.png) **What works** (all Playwright-driven against real interactions): boards / lists / cards CRUD, **drag-and-drop** cross-list moves with persistence, a markdown card modal with live preview, labels / due dates / checklists, keyboard shortcuts, and **live search/filter** — below, typing `drag` narrows to the matching card in real time: ![FLUX live search](assets/flux-search.png) **Analysis.** Over five phases the model needed only **two** corrections, each caught by Playwright and fixed by the model: an undeclared `dragMoveOffsetX` (1-line) and a search UI that was built but not yet wired to the filter logic. ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## ◆ Architecture details | Field | Value | |---|---| | `general.architecture` | `qwen35` (hybrid linear + full attention) | | Parameters | ~27B (size label `27B`) | | Layers (`block_count`) | 64 | | Hidden size | 5120 | | FFN size | 17408 | | Attention heads / KV heads | 24 / 4 (GQA) | | Full-attention interval | every 4th layer (`full_attention_interval = 4`); the rest are linear/DeltaNet | | Linear-attn (SSM) | conv kernel 4, state 128, groups 16, inner size 6144, time-step rank 48 | | Key/Value length | 256 | | Vocab | 248,320 | | Native context | 262,144 | | RoPE | freq base 1e7, mRoPE sections `[11, 11, 10]`, partial rotary (dim 64) | | MTP / NextN head | present in parents (copied from V2) | | Precision | **`BF16`** (unquantized, file type 32) — full-precision reference weights | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## ◆ Usage ### ▌ llama.cpp (recommended) Validated inference config (single-stream, thinking on): ```bash llama-server \ -m Qwopus3.6-27B-Fusion-BF16.gguf \ -c 98304 \ # 96K context; native is 262K. BF16 weights are ~55GB — size VRAM/RAM accordingly --parallel 1 \ # SINGLE stream — validated single-stream -ngl 999 \ # all layers on GPU --cache-type-k q8_0 --cache-type-v q8_0 \ --reasoning on --jinja \ # thinking mode + the baked-in chat template --host 0.0.0.0 --port 8001 ``` Recommended sampling: `temperature 0.85–1.0 top_p 0.9`. **Temperature — run it hot.** Best capability lands at **temp 0.85–1.0** (the finetune's natural operating point, and consistent with the agentic/SWE results). Then the surprising part: **the merge itself appears to have repaired much of the low-temperature instability** of the parent finetunes — Fusion stays clean at temps where the bases loop and degrade (**4/4 termination at *both* 0.2 and 0.9**, 10/10 across 0.5–0.9). So, *unlike its parents*, you **can** run it cooler for more determinism if you want, without the looping. Exactly why merging improves temperature stability isn't fully understood, but it's consistent across every test we ran. **High temperature still yields the best results.** Thinking mode is **on** by default and is where the model is strongest. ### ▌ As a coding backend (Claude Code / OpenAI-compatible clients) The `llama-server` endpoint is OpenAI-compatible and has driven multi-file app builds end-to-end. If you point **Claude Code** at it, use a proxy that does **native Anthropic↔OpenAI tool translation** (text-encoded tools stall the agent). ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## ◆ Evaluation — full results Benchmarks were run on the **`Q4_K_M`** quant (thinking-on unless noted); the BF16 weights in this repo are the full-precision source those results derive from and should meet or exceed them. **Capability:** HumanEval 94.5% · MBPP 87.9% · GSM8K 95.0% · MATH-500 62% (64.7% subset) · IFEval-strict 82.8%. **Stability** (the axis that separated this project's winners from losers): | Test | Result | |---|---| | Termination @ temp 0.2 / 0.9 | 4/4 / 4/4 clean | | Temperature sweep 0.5–0.9 | 10/10 clean (9/10 at 0.1 & 1.0) | | Seed variance — HumanEval / MATH | std 0.0 / 2.36 | | Long-context needle + termination | perfect to 60K tokens, zero looping | **Agentic (SWE-bench Verified, astropy slice, thinking-on):** Fusion **7/15 (47%)** > equal-weight DARE-TIES **5/15** > DUS-38B frankenmerge **3/15**. ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## ◆ Limitations - **Not yet rigorously evaluated.** No large-scale safety alignment / RLHF / red-teaming. Inherits biases and limits of the Qwen base and both parent finetunes. - The astropy **+2 (7/15 vs 5/15) is within noise** on n=15 — the geometry approach is validated as *sound and directionally better*, not a large agentic jump. - On the hardest agentic instances the model can over-deliberate and hit the generation cap without submitting; capping `max_tokens` (~8K) mitigates this. - Validated primarily at `Q4_K_M`; other quants should work but aren't separately benchmarked. > **Found a problem? →** please reach out on X: **[@kylehessling1](https://x.com/kylehessling1)**. ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## ◆ Method credit & acknowledgments The path to this model was not a straight line, and the credit is honest about that: - **[Jackrong](https://huggingface.co/Jackrong) — base-model builder, collaborator, and good friend.** This fusion stands entirely on his work: he built both parent finetunes ([`Qwopus3.6-27B-v2`](https://huggingface.co/Jackrong/Qwopus3.6-27B-v2) and [`Qwopus3.6-27B-Coder`](https://huggingface.co/Jackrong/Qwopus3.6-27B-Coder)) that this merge fuses. Beyond the models, his counsel and contributions throughout were **priceless** — this project simply does not exist without him. **Go follow his work → [huggingface.co/Jackrong](https://huggingface.co/Jackrong).** - The **initial naive approaches** — a 38B depth-upscaled (DUS) frankenmerge and an equal-weight DARE-TIES merge from the shared base — were **proposed by Fable**. Both underdelivered. In hindsight, those plans conspicuously **omitted the one nuance that turned out to be decisive**: the *lineage* relationship between the parents (that the coder is a descendant of V2). Without that nuance, an equal-weight merge is essentially guaranteed to double-count V2's delta and import the destabilizing early-layer coding delta at full strength — which is exactly what went wrong. - **Grok 4.5** was **instrumental in the final, working merge.** Its analysis surfaced the parent→descendant lineage from the weight geometry and motivated the **layer-weighted delta-scale** recipe (protect early reasoning, inject coding late) that this model is built on. ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## ◆ Lineage / provenance ``` Qwen/Qwen3.6-27B (shared base) ├── Qwopus3.6-27B-v2 (reasoning finetune) ──┐ └── Qwopus3.6-27B-Coder (code finetune, │ ← descends from V2 (proj→1.0 in late layers) itself downstream of V2) │ ▼ Qwopus3.6-27B-Fusion = V2 + α(L)·(Coder − V2), α: 0.12 → 0.48 by depth ``` ## ◆ License A merge of derivatives of `Qwen/Qwen3.6-27B`; usage is governed by the **Qwen base model license**. The parent finetunes are personal research models. No third-party training data is redistributed here — only merged weights and the merge script. Comply with the upstream Qwen license. ## ◆ Reproduction ```bash python merge_layerweighted.py --out ./Qwopus3.6-27B-Fusion --a-early 0.12 --a-late 0.48 # then convert + quantize with llama.cpp: convert_hf_to_gguf.py → llama-quantize ... Q4_K_M ``` If you build on the *method* (geometry-grounded, layer-weighted task-vector merging of a parent/descendant finetune pair), a mention of **"FusionThink layer-weighted delta-scale merge"** is appreciated. ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Built and evaluated locally on a single RTX 5090. Model produced by Kyle Hessling (@kylehessling1).