README.md

---
license: other
license_name: qwen
license_link: https://huggingface.co/Qwen/Qwen3.6-27B/blob/main/LICENSE
language:
- en
tags:
- merge
- task-vector
- dare-ties
- layer-weighted
- qwen3
- qwen35
- reasoning
- code
- gguf
- llama.cpp
- experimental
base_model:
- Qwen/Qwen3.6-27B
pipeline_tag: text-generation
library_name: gguf
---

<div align="center">

<img src="assets/hero-banner.png" alt="Qwopus3.6-27B Fusion — one 27B that reasons and codes, no skill lost" width="100%">

<h3>▶ <a href="https://huggingface.co/spaces/KyleHessling1/ORBITAL">PLAY <i>ORBITAL</i> — a game this model built, live in your browser</a> ◀</h3>

</div>

> **Research preview.** This model attempts to combine the **reasoning** capability of
> `Qwopus3.6-27B-v2` with the **code-execution** capability of `Qwopus3.6-27B-Coder` — in a single 27B
> model, *without significant loss to either* skill. In practice it behaves like a production model
> with both capabilities fused, but it has **not** yet been through a full rigorous evaluation. If you
> find issues, please reach out on X — **[@kylehessling1](https://x.com/kylehessling1)**.

- **Format:** GGUF, `Q4_K_M` (~16.5 GB) — runs on a single 24–32 GB GPU via `llama.cpp`.
- **Architecture:** `qwen35` hybrid (linear-attention + periodic full-attention), 27B params, 64 layers.
- **Native context:** 262,144 tokens. Verified clean (needle + termination) to 60K.
- **Modes:** thinking (reasoning) **on** by default; strong agentic/coding behavior.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

## ◆ TL;DR

| | |
|---|---|
| **Base / anchor** | `Qwen/Qwen3.6-27B` (both parents descend from it) |
| **Parent A — reasoning** | `Qwopus3.6-27B-v2` — donates the merge base + embeddings/head/norms/MTP |
| **Parent B — code** | `Qwopus3.6-27B-Coder` — delta source, injected with depth-increasing weight |
| **Merge math** | `W(L) = V2 + α(L)·(Coder − V2)`, α linear ramp **0.12 (early) → 0.48 (late)** |
| **Frozen (copied from A)** | `embed_tokens`, `lm_head`, all `norm`, `mtp`/NextN, any vision tensors |

**Headline results** (Q4_K_M, thinking-on unless noted):

| Capability | Score | Stability | Score |
|---|---|---|---|
| HumanEval | **94.5%** | Termination @ temp 0.2 & 0.9 | **4/4 clean** |
| MBPP | **87.9%** | Temp sweep 0.5–0.9 | **10/10 clean** |
| GSM8K | **95.0%** | Seed variance (HumanEval) | **std 0.0** |
| MATH-500 | **62–64.7%** | Long-context needle+term @ 60K | **perfect** |
| IFEval (strict) | **82.8%** | SWE-bench Verified (astropy, 0:15) | **7/15 (47%)** |

> Against the alternatives on the same astropy slice: **Fusion 7/15** > equal-weight DARE-TIES **5/15** > DUS-38B frankenmerge **3/15**.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

## ◆ Where it shines — the one-stop creative programming agent

> **Best used as a single, closed-loop agentic creative programmer.** In a *build → run → self-fix*
> loop, Fusion is at its strongest as **one agent that plans, writes, and debugs on its own** — the
> reasoning⊕coding balance lets it close the loop by itself (see the ORBITAL / FLUX demos below,
> where it root-caused and fixed its own bugs from Playwright error reports).
>
> This is a different niche than its parents. The **`Coder`** model is excellent for **coding under
> active steering**, or as a **solid orchestration model** driving other tools/agents. **Fusion**, by
> contrast, is the ideal **one-stop creative builder** — you hand it a goal and it carries the whole
> creative-programming loop end-to-end.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

## ◆ Why I chose to make this merge

Two finetunes of the same base diverge into complementary specialists: one gets better at
multi-step reasoning, the other at writing code — but each pays for it. In particular the **coding
finetune becomes unstable in thinking mode** (it loops / degrades at low temperature). The goal was
a single model that keeps the reasoner's stability *and* the coder's skill.

My naive approaches, one of them eroneously proposed by Claude Fable, average the two, or run an equal-weight DARE-TIES merge from the common base. These alternatives underperformed and the geometry explains why.

### ▌ The lineage

Because the coder model descended from the v2 version, the coders additions can be evaluated against the v2 by analyzing the delta. Projecting the coding
delta onto the reasoning delta reveals:

```
proj(ΔCoder · ΔV2 / |ΔV2|²)   ≈ 0.4  in early layers   →   ≈ 1.0  in late layers
```

That ramp to 1.0 means **late layers of the coder already contain V2's full reasoning delta**. And:

- The coding delta is **~1.8× the magnitude** of the reasoning delta.
- In **early layers** the coding delta is **anti-aligned** with reasoning (`cos < 0`) — the coder
  *partially reverted* V2's early-layer reasoning/control representations. **That anti-alignment is
  the measured source of the coder's thinking-mode instability.**

An equal-weight merge (both task vectors added from the base) therefore commits two errors:
1. It **double-counts V2's reasoning delta** (the coder already contains it), and
2. It **imports the full early-layer destabilizing coding delta** at 100% strength.

### ▌ The fix

Because the coder sits "downstream" of V2, the correct anchor is **V2 itself**, not the shared base
— this avoids the double-count. Then, instead of a constant fraction of the coding delta everywhere,
inject it with a **depth-increasing schedule**:

- **Early layers (α ≈ 0.12):** inject *little* coding delta → **protect** V2's reasoning/control
  reps exactly where the coder damaged them.
- **Late layers (α ≈ 0.48):** inject *more* coding delta → add coding skill where the two parents
  already agree (proj → 1.0) and it's safe.

The shape of the α ramp is dictated by the geometry, not hand-tuning. Embeddings, LM head, norms,
and the MTP head are identical across all three models (delta ≈ 0), so they're copied verbatim from
V2 rather than merged.

<div align="center">
<img src="assets/geometry.png" alt="Weight geometry — coder descends from V2, its delta is 1.8x and anti-aligned in early layers; the alpha ramp injects coding 0.12 early to 0.48 late" width="100%">
</div>

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

## ◆ How it was built

**Merge.** The parents use Qwen's **hybrid attention schedule** (most layers linear/DeltaNet; every
4th layer full softmax attention). Off-the-shelf `mergekit` **cannot** merge this architecture — it
templates `linear_attn.*` onto every layer and errors on the full-attention layers. So the merge is
a **custom streaming task-vector routine** operating directly on the (identical) tensor keys, immune
to arch templating. The full, reproducible script ships in this repo as
[`merge_layerweighted.py`](./merge_layerweighted.py):

```python
def alpha(L, a_early, a_late):
    return a_early + (a_late - a_early) * (L / (NLAYERS - 1))   # smooth linear ramp by depth

# per decoder tensor at layer L:
out = (v2 + alpha(L, 0.12, 0.48) * (coder - v2)).to(torch.bfloat16)
# embeddings / lm_head / norms / mtp / vision: copied from V2 verbatim
```

**Quantize.** The bf16 merge was converted to GGUF and quantized to **`Q4_K_M`** with `llama.cpp`
(851 GGUF tensors, 16.5 GB).

**Evaluate.** Every candidate was gated at **both** temp 0.2 (a termination stress test) and temp
0.9 (the finetune operating point) — "clean only at high temp" is riding sampling entropy, not
stability. This model is clean at both.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

## ◆ Demonstrations — driving a local agentic coding loop

Every artifact below was generated by **this model** running as the backend of an agentic coding
loop (OpenCode / Claude Code), building single-file HTML/JS apps end-to-end, then **verified with
headless Playwright**. What stood out isn't just that it writes polished games — it's that when the
Playwright harness handed back error reports, **the same model root-caused and fixed its own bugs**
with precise, correct diagnoses (excerpts below).

### 🚀 ORBITAL — physics space-arcade (thinking-on)

*Setup:* "Build a complete, physics-driven, visually stunning arcade space game as a single
self-contained index.html — gravity is the weapon." *Outcome:* **built and runs cleanly, zero
console errors**, real 2-body gravity, particle trails, wave HUD, and it even self-balance-tuned its
own difficulty. 1,150 lines.

<div align="center">
<h3>▶ <a href="https://huggingface.co/spaces/KyleHessling1/ORBITAL">PLAY IT LIVE → huggingface.co/spaces/KyleHessling1/ORBITAL</a> ◀</h3>
<i>(runs entirely in your browser — the exact single file this model wrote)</i>
</div>

| Title screen | Live gameplay (Wave 1) |
|---|---|
| ![ORBITAL title](assets/orbital-title.png) | ![ORBITAL gameplay](assets/orbital-gameplay.png) |

The same loop also produced three more complete, self-contained games — **PRISM DEFENSE** (neon
tower defense, 1,074 lines), **NEON SERPENT** (synthwave Snake with smooth interpolated movement,
657 lines), and **STARLIGHT SURVIVORS** (survivors-style auto-shooter, 555 lines). Each was built,
Playwright-tested, and where the harness surfaced a bug the model root-caused and fixed it itself
(e.g. a tower-targeting `bestDist = Infinity` that should have been `-Infinity`; three weapon
upgrades missing a `fireTimer` init that made `fireTimer -= dt` evaluate to `NaN`). The impressive
part is the quality of the model's own debugging inside a *build → error report → self-fix* loop,
not flawless one-shot output.

### 📋 FLUX — a full Trello-style Kanban dashboard (the strongest capability demo)

The most demanding build: a complete **single-file Kanban app** (`~3,373 lines`, no backend,
localStorage persistence), assembled by the model over **five steered phases**, each a fresh session
that read a spec + the current file and extended it — Playwright-verified between phases.

![FLUX board](assets/flux-board.png)

**What works** (all Playwright-driven against real interactions): boards / lists / cards CRUD,
**drag-and-drop** cross-list moves with persistence, a markdown card modal with live preview,
labels / due dates / checklists, keyboard shortcuts, and **live search/filter** — below, typing
`drag` narrows to the matching card in real time:

![FLUX live search](assets/flux-search.png)

**Analysis.** Over five phases the model needed only **two** corrections, each caught by Playwright
and fixed by the model: an undeclared `dragMoveOffsetX` (1-line) and a search UI that was built but
not yet wired to the filter logic.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

## ◆ Architecture details

| Field | Value |
|---|---|
| `general.architecture` | `qwen35` (hybrid linear + full attention) |
| Parameters | ~27B (size label `27B`) |
| Layers (`block_count`) | 64 |
| Hidden size | 5120 |
| FFN size | 17408 |
| Attention heads / KV heads | 24 / 4 (GQA) |
| Full-attention interval | every 4th layer (`full_attention_interval = 4`); the rest are linear/DeltaNet |
| Linear-attn (SSM) | conv kernel 4, state 128, groups 16, inner size 6144, time-step rank 48 |
| Key/Value length | 256 |
| Vocab | 248,320 |
| Native context | 262,144 |
| RoPE | freq base 1e7, mRoPE sections `[11, 11, 10]`, partial rotary (dim 64) |
| MTP / NextN head | present in parents (copied from V2) |
| Quant | `Q4_K_M` (file type 15), quant version 2 |

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

## ◆ Usage

### ▌ llama.cpp (recommended)

Validated inference config (single-stream, thinking on):

```bash
llama-server \
  -m tv-lw-balanced-Q4_K_M.gguf \
  -c 98304 \                 # 96K context; native is 262K but 96K fits comfortably on 32GB
  --parallel 1 \             # SINGLE stream — validated single-stream
  -ngl 999 \                 # all layers on GPU
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --reasoning on --jinja \   # thinking mode + the baked-in chat template
  --host 0.0.0.0 --port 8001
```

Recommended sampling: `temperature 0.85–1.0  top_p 0.9`.

**Temperature — run it hot.** Best capability lands at **temp 0.85–1.0** (the finetune's natural
operating point, and consistent with the agentic/SWE results). Then the surprising part: **the merge
itself appears to have repaired much of the low-temperature instability** of the parent finetunes —
Fusion stays clean at temps where the bases loop and degrade (**4/4 termination at *both* 0.2 and
0.9**, 10/10 across 0.5–0.9). So, *unlike its parents*, you **can** run it cooler for more
determinism if you want, without the looping. Exactly why merging improves temperature stability
isn't fully understood, but it's consistent across every test we ran. **High temperature still
yields the best results.** Thinking mode is **on** by default and is where the model is strongest.

### ▌ As a coding backend (Claude Code / OpenAI-compatible clients)

The `llama-server` endpoint is OpenAI-compatible and has driven multi-file app builds end-to-end. If
you point **Claude Code** at it, use a proxy that does **native Anthropic↔OpenAI tool translation**
(text-encoded tools stall the agent).

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

## ◆ Evaluation — full results

All at `Q4_K_M`, thinking-on unless noted.

**Capability:** HumanEval 94.5% · MBPP 87.9% · GSM8K 95.0% · MATH-500 62% (64.7% subset) ·
IFEval-strict 82.8%.

**Stability** (the axis that separated this project's winners from losers):

| Test | Result |
|---|---|
| Termination @ temp 0.2 / 0.9 | 4/4 / 4/4 clean |
| Temperature sweep 0.5–0.9 | 10/10 clean (9/10 at 0.1 & 1.0) |
| Seed variance — HumanEval / MATH | std 0.0 / 2.36 |
| Long-context needle + termination | perfect to 60K tokens, zero looping |

**Agentic (SWE-bench Verified, astropy slice, thinking-on):** Fusion **7/15 (47%)** >
equal-weight DARE-TIES **5/15** > DUS-38B frankenmerge **3/15**.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

## ◆ Limitations

- **Not yet rigorously evaluated.** No large-scale safety alignment / RLHF / red-teaming. Inherits
  biases and limits of the Qwen base and both parent finetunes.
- The astropy **+2 (7/15 vs 5/15) is within noise** on n=15 — the geometry approach is validated as
  *sound and directionally better*, not a large agentic jump.
- On the hardest agentic instances the model can over-deliberate and hit the generation cap without
  submitting; capping `max_tokens` (~8K) mitigates this.
- Validated primarily at `Q4_K_M`; other quants should work but aren't separately benchmarked.

> **Found a problem? →** please reach out on X: **[@kylehessling1](https://x.com/kylehessling1)**.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

## ◆ Method credit & acknowledgments

The path to this model was not a straight line, and the credit is honest about that:

- **[Jackrong](https://huggingface.co/Jackrong) — base-model builder, collaborator, and good friend.**
  This fusion stands entirely on his work: he built both parent finetunes
  ([`Qwopus3.6-27B-v2`](https://huggingface.co/Jackrong/Qwopus3.6-27B-v2) and
  [`Qwopus3.6-27B-Coder`](https://huggingface.co/Jackrong/Qwopus3.6-27B-Coder)) that this merge fuses.
  Beyond the models, his counsel and contributions throughout were **priceless** — this project simply
  does not exist without him. **Go follow his work → [huggingface.co/Jackrong](https://huggingface.co/Jackrong).**
- The **initial naive approaches** — a 38B depth-upscaled (DUS) frankenmerge and an equal-weight
  DARE-TIES merge from the shared base — were **proposed by Fable**. Both underdelivered. In
  hindsight, those plans conspicuously **omitted the one nuance that turned out to be decisive**: the
  *lineage* relationship between the parents (that the coder is a descendant of V2). Without that
  nuance, an equal-weight merge is essentially guaranteed to double-count V2's delta and import the
  destabilizing early-layer coding delta at full strength — which is exactly what went wrong.
- **Grok 4.5** was **instrumental in the final, working merge.** Its analysis surfaced the
  parent→descendant lineage from the weight geometry and motivated the **layer-weighted delta-scale**
  recipe (protect early reasoning, inject coding late) that this model is built on.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

## ◆ Lineage / provenance

```
Qwen/Qwen3.6-27B  (shared base)
├── Qwopus3.6-27B-v2      (reasoning finetune)  ──┐
└── Qwopus3.6-27B-Coder   (code finetune,        │  ← descends from V2 (proj→1.0 in late layers)
     itself downstream of V2)                    │
                                                  ▼
        Qwopus3.6-27B-Fusion = V2 + α(L)·(Coder − V2),  α: 0.12 → 0.48 by depth
```

## ◆ License

A merge of derivatives of `Qwen/Qwen3.6-27B`; usage is governed by the **Qwen base model license**.
The parent finetunes are personal research models. No third-party training data is redistributed
here — only merged weights and the merge script. Comply with the upstream Qwen license.

## ◆ Reproduction

```bash
python merge_layerweighted.py --out ./Qwopus3.6-27B-Fusion --a-early 0.12 --a-late 0.48
# then convert + quantize with llama.cpp: convert_hf_to_gguf.py  →  llama-quantize ... Q4_K_M
```

If you build on the *method* (geometry-grounded, layer-weighted task-vector merging of a
parent/descendant finetune pair), a mention of **"FusionThink layer-weighted delta-scale merge"** is
appreciated.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

<div align="center">
<i>Built and evaluated locally on a single RTX 5090. Model produced by Kyle Hessling
(<a href="https://x.com/kylehessling1">@kylehessling1</a>).</i>
</div>