docs/features/rys_layer_duplication.md

RYS runtime layer duplication (weight-shared "Repeat-Your-Self")

Status: DESIGN + PROTOTYPE (2026-07-12). Parser/expander prototyped and validated; edit surface mapped; MVP scoped. No source landed yet.

Child of MASTER_PLAN.md. Additive soft-fork feature — new load-time flag, per-arch surgical hooks in the graph loop + KV alloc.

1. What this is

David Noel Ng's RYS ("Repeat Your Self", rys-ii) shows that re-running a contiguous block of middle transformer layers — no weight changes, no training — measurably improves a model (the layers duplicated are the ones operating in the format-agnostic "reasoning" band; encoding/decoding boundary layers must not be touched). For Qwen3.5-27B the Pareto-optimal blocks are tiny: (33,34) (+1 layer) already captures most of the EQ gain, up through (26,34) (+8) for the best absolute score.

The naive way to ship a RYS variant is a physically merged GGUF (mergekit passthrough) — but that doubles the duplicated layers' weight bytes on disk and in VRAM. dnhkng is instead working with TurboDerp on pointer-based duplication in ExLlamaV3: the repeated layers share the weight tensors with their originals, so "no additional VRAM is consumed for the parameters themselves — you only pay extra for the compute time and KV cache of the additional forward passes."

This feature ports that pointer-shared idea to opencoti llamafile, driven purely by a load-time flag. No new GGUF, no re-quant, quant-agnostic:

llamafile --server -m Qwen3.5-27B-*.gguf --repeat-layers 26,34   # RYS-XL, +8 layers
llamafile --server -m Qwen3.5-27B-*.gguf --repeat-layers 33,34   # RYS-S,  +1 layer

Stock llama.cpp has no runtime layer-repeat; this is a genuine opencoti differentiator and fits our "additive, load-time" ethos.

2. Mechanism: a layer execution plan (eff → src)

llama.cpp's per-architecture graph builder runs a residual loop for (il = 0; il < n_layer; ++il) that (a) reads weights from model.layers[il] and (b) uses il to index the KV cache, RoPE, and per-layer hparams. We replace the identity iteration with a plan: a vector of source layer indices to execute in order.

• layer_plan[eff] = src — for each effective position eff, which source layer's weights to run. Default (no flag) = identity [0,1,…,n_layer-1].
• A RYS block (i,j) (half-open, matching the article) inserts source indices i…j-1 a second time right after their first pass.
• n_layer_eff = layer_plan.size() — the effective layer count. This is what the KV cache, RoPE loop, and residual loop use. The GGUF's n_layer (the source weight count) is unchanged — weight loading is untouched.

Forward loop, conceptually:

for (int eff = 0; eff < n_layer_eff; ++eff) {
    const int src = layer_plan[eff];
    const auto & layer = model.layers[src];   // WEIGHTS: shared by pointer, zero extra VRAM
    cur = build_norm(inpL, layer.attn_norm, …, eff);
    cur = build_layer_attn(inp, cur, inp_pos, …, /*kv_il=*/eff);   // KV/cache index: eff (distinct slot)
    …
    inpL = cur;
}

The duplicated pass reads the same weights but writes its own K/V at the same token positions — exactly the RYS semantics.

2a. Prototype (validated)

The plan expander is prototyped and unit-tested against every config in the article's Qwen3.5-27B Pareto table (reference: scratchpad/rys_plan.py; the C++ will mirror it):

spec	blocks	n_eff (from 64)	extra	article
33,34	(33,34)	65	+1	+1 ✓
31,34	(31,34)	67	+3	+3 ✓
30,35	(30,35)	69	+5	+5 ✓
26,34	(26,34)	72	+8	+8 ✓
24,35	(24,35)	75	+11	+11 ✓
29,34	(29,34)	69	+5	+5 ✓

(26,34) expands to …,24,25,[26,27,28,29,30,31,32,33],[26,27,28,29,30,31,32,33],34,35,… — the seam is the duplicated 26..33. Overlapping blocks are rejected in the MVP (disjoint-ascending only; the Pareto winners are all single blocks).

3. Parameter grammar

--repeat-layers <spec>

• <spec> = one or more blocks separated by ;.
• Each block is i,j or i-j = the half-open range [i,j) of source transformer-layer indices to duplicate in place. i,j matches the article's (i,j) notation for copy-paste from the HF configs.
• Examples: 33,34 · 26,34 · 43,45;28,34 (compose two disjoint blocks).
• Validation (fail-fast at load): 0 ≤ i < j ≤ n_transformer_layers; blocks disjoint; the MTP/nextn_predict_layers head is never duplicable.

Also plumbed through the opencoti TS adapter (buildServerArgs) as repeatLayers?: string so opencode configs can set it.

4. Memory / cost model

• Params: zero extra VRAM — duplicated layers point at the same model.layers[src] tensors.
• KV cache: grows by the extra passes — n_layer_eff/n_layer. E.g. (26,34) on a 64-layer model = +8 → +12.5% KV. Modest; the article calls it out explicitly. (This is why RYS-S (33,34), +1.5% KV, is the low-overhead pick.)
• Compute: +one forward pass per duplicated layer per token — same ratio as KV. The win is quality-per-token, paid in tokens/s.

5. What is provably unaffected

• Weight loading — untouched; n_layer (source) drives create_tensor.
• RoPE / positions — positions come from the batch, not the layer. A source layer run at two eff positions sees the same token positions; only its KV slot differs. No self-extend-style position hacking.
• Off (no flag) — layer_plan = identity ⇒ byte-identical to stock. This is the primary regression gate.

6. Composition with opencoti per-layer features

Everything opencoti indexes by layer must consume the effective index (or map eff→src for source-keyed attributes). Scoped incrementally:

feature	keyed by	MVP posture
plain f16 / scalar-quant KV	eff slot	MVP — works directly
RoPE / positions	batch pos	unaffected
iSWA (Gemma sliding/global per layer)	src (is_swa(il))	Gemma phase — map eff→src
DCA (all-KV)	per-layer	later — gate off under --repeat-layers first, then make plan-aware
rolling-KV (window_cells[il])	eff budget	later — size by n_layer_eff
PolyKV / sparse-attn	per-layer	later
MTP / NextN head	excluded	head never duplicated; spec path unchanged

Survey nuance (important): DCA, rolling-KV (window_cells[il]), PolyKV, and sparse-attn are all keyed through map_layer_ids / per-physical-layer structs — so they auto-inherit eff-correctness the moment the :927 alloc loop (§8b#4) is eff-length. The residual risk is not the accessors; it's the sizing loops that still say hparams.n_layer and would under-allocate for eff slots: n_layer_kv(), opencoti_compute_resident_window_cells (kv-cache:249/:772), and the sparse block-sel reserve 8*n_layer_kv (kv-cache:879). Plus DCA's is_swa(il) in gemma4:217 needs src.

MVP rule: ship --repeat-layers composing only with plain/quant KV on the dense arch first; hard-error at load if combined with DCA / rolling-KV / sparse / MTP until each sizing loop is made eff-aware (explicit follow-ups). This prevents silent under-allocation, which would corrupt rather than error.

7. Per-architecture scope

Phase 1 — Qwen3.5 / Qwen3.6 dense (MVP target)

Uniform layers (no iSWA, uniform head counts/rope) ⇒ eff→src is trivial for everything except KV-slot allocation. nextn_predict_layers (MTP head) excluded from the plan. Target models: Qwen3.5-27B dense, Qwen3.6 dense variants. Forward loop: models/qwen35.cpp (also covers the dense path), models/qwen3.cpp.

Phase 2 — Gemma-4 dense 12B and 31B

Adds the iSWA wrinkle: alternating sliding-window / global layers, per-layer query scaling / attn logit-softcap. A duplicated layer must inherit its source layer's SWA type and scaling ⇒ per-layer hparam lookups map eff→src. The KV cache's SWA/global split (kv_local_layer, the iSWA cache) must size by n_layer_eff with the src-derived SWA pattern. Forward loop: models/gemma4.cpp. Targets: gemma-4-12B-it dense, gemma-4-31B-it dense.

(Gemma-4 A4B is MoE — out of scope for the dense MVP; RYS on MoE is a separate question about whether per-expert routing survives block re-traversal.)

8. Edit surface

Line-anchored from the source survey (2026-07-12). All paths under vendors/sources/llamafile/llama.cpp/.

Governing finding: in this fork the il handed to build_attn(...) is used for BOTH weight lookup (model.layers[il] in the model .cpp) AND KV-slot lookup (map_layer_ids[il]). RYS decouples them: weights + architecture props (is_swa, rope-freq, n_head, n_rot) index by src; the KV-slot index handed to build_attn is eff. For Qwen dense this is a no-op distinction outside KV (uniform layers, is_swa always false), so build_attn(eff) is clean. For Gemma the iSWA sub-cache choice must be src-driven (see §8b).

8a. C-ABI plumbing caveat (load-bearing)

struct llama_context_params (include/llama.h:336) crosses the C ABI — it cannot carry a std::vector. So:

• include/llama.h:336 — add a compact int spec (uint32_t repeat_start, repeat_end, repeat_count, or a small CSV string).
• src/llama-cparams.h:9 — add std::vector<int> layer_plan; to llama_cparams (plain C++ struct, vector OK). Insert near the opencoti extension fields (69-80).
• src/llama-context.cpp:~133 — expand the compact spec into cparams.layer_plan here (where hparams.n_layer is known); default unset = identity 0..n_layer-1; set n_layer_eff = layer_plan.size().

8b. Minimal MVP edit set — Qwen dense (plain KV)

#	site	change	invasiveness
1	llama-cparams.h:9	add layer_plan vector	trivial
2	include/llama.h:336	compact int spec (C-ABI)	trivial
3	llama-context.cpp:~133	expand spec → layer_plan; n_layer_eff	trivial
4	llama-kv-cache.cpp:927 (+ sizing :165/:278/:451/:478)	iterate n_layer_eff; size each eff slot from src=plan[eff] (n_embd_k_gqa(src), has_kv(src)); map_layer_ids[eff]=layers.size()	moderate
5	models/qwen3.cpp:76 (·qwen2.cpp:76·qwen3moe.cpp:86)	loop over eff; src=plan[eff]; model.layers[il]→[src]; build_attn(…,eff); out-ids guard eff==n_layer_eff-1	moderate
6	common/arg.cpp:~2780	add_opt("--repeat-layers") (mirror --override-kv CSV)	trivial
7	common/common.cpp:~1660	map common_params range → llama_context_params spec	trivial

qwen35.cpp:169 — the plan is built over [0, n_transformer_layers) where n_transformer_layers = n_layer - nextn_predict_layers (:167), so the MTP head is already excluded by construction; never duplicate a nextn block.

KV precedent: llama-kv-cache.cpp already tolerates a non-identity map_layer_ids — the il_share path (:946-963, bug-858 dual-ctx MTP) and il_reuse (:1265-1283, SWA reuse). Both share physical slots; RYS is the inverse (allocate more, distinct slots), so no new indirection type is needed — just an eff-length domain on the :927 alloc loop. The 30+ read accessors (get_k/cpy_k via mli_at_checked(map_layer_ids, il, …)) are unchanged as long as the graph passes eff and every eff has a map entry.

8c. Additional edits — Gemma-4 dense (iSWA), the invasive part

site	change
models/gemma4.cpp:194	loop rewrite splitting src/eff: is_swa(src), get_rope_freq_base/scale(cparams,src) (:201-202), n_rot(src), layers[src].rope_freqs, f_attention_scale by src; feed eff only to build_attn (:269-283) + KV routing
gemma4.cpp:164 iSWA dual-cache	build_attn_inp_kv_iswa() + kv-cache filter/reuse must assign each eff slot to base-vs-swa sub-cache by is_swa(plan[eff]) — pass a plan-aware filter into the iSWA cache ctor (the invasive step: it currently reads swa_layers[il])
llama-model.cpp:1929/1933/1937	get_rope_freq_base/scale/factors only ever called with src (they branch on is_swa)

8d. hparams helpers (index by src)

llama-hparams.cpp: is_swa(il) :208, n_head(il) :30, n_head_kv(il) :38, n_rot(il) :65, n_embd_k_gqa/v_gqa(il) :103/:109, has_kv(il) :231, n_layer_kv() :250, swa_layers[] (hparams.h:141). No signature changes — the rule is "call these with src in the loop." RoPE positions come from the batch (llama-graph.cpp:161), not the layer, so a src layer run at two eff positions is correct with no position hacking.

9. Validation plan

Correctness is mechanism correctness (the plan re-traverses exactly the intended layers) — RYS quality is the user's block choice, not ours to prove.

1. Off = identity — --repeat-layers absent ⇒ logits byte-identical to stock (teacher-forced, real_frac=0). Primary regression gate.
2. Plan structural — boot log dumps n_layer_eff + the eff→src plan; assert it matches the spec (e.g. (26,34) → 72 layers, seam at 34/42).
3. Deterministic divergence — (33,34) vs base produces a deterministic, non-degenerate logit change (greedy, same GGUF) — proves the duplicated pass actually runs and feeds forward, not a no-op.
4. KV integrity — niah retrieval stays 100 at a short ctx with a duplicated block (the extra KV slots are wired correctly; no cross-slot aliasing).
5. Known-win spot check — reproduce the article's direction: a small EQ/math probe should move in the reported direction for (33,34) / (26,34) on Qwen3.5-27B (sanity that we duplicated the reasoning band, not garbage).
6. Gemma iSWA — the src-derived SWA pattern is honored (duplicated global layer stays global); niah 100 + coherent decode.

Correctness always via logit-equiv / niah — never greedy-needle-as-proof.

10. Risks / caveats

• KV growth is real (§4) — surface it in the boot log; the TS adapter should account for it in VRAM budgeting.
• iSWA mapping — feared to be the main Gemma risk, but verified correct (see §10a): a middle SWA layer duplicates coherently, so the dual-cache eff→src wiring is sound. What actually breaks is boundary layers, independent of SWA.
• MoE out of scope for dense MVP — but the iSWA path was validated on both dense (31B, 60 layers) and MoE (A4B, 26B-A4B-128e) with identical behaviour.
• Upstream syncability — new flag + per-arch loop hooks tagged opencoti-hook: rys-layer-dup and registered in UPSTREAM_SYNC.md; the layer_plan indirection is small and localized to keep future bumps cheap.

10a. Boundary-layer fragility (bug-2164) — a MODEL property, not an engine bug

Duplicating the first or last transformer layers reliably produces incoherent output. This is the well-documented franken-merge / passthrough self-merge fragility (mergekit passthrough, SOLAR depth-upscaling all DROP the first/last m layers), NOT an engine defect. The engine duplicates any layer faithfully.

Decisive 2×2 (2026-07-13, native google_gemma-4-26B-A4B-it-Q4_K_M, 3090, CHAT endpoint — the raw /completion greedy path is INVALID on this thinking model: its answer lands in reasoning_content, so even RYS-off looks like garbage there). Garbage is determined entirely by boundary-ness, independent of SWA/global:

	SWA layer	GLOBAL layer
middle (L4, L10 / L5, L17)	coherent ✓	coherent ✓
boundary (L0, L28 / L29)	garbage ✗	garbage ✗

• Gemma-4 A4B coherent band ≈ L4..L24; L0–L3 (early) and L25–L29 (late) break.
• Qwen3-8B has a wider tolerant band — only its last layer (L35) breaks.
• Every attention output + final logit stays finite (no NaN) even under the garbage config → finite-but-wrong = model-level, not a numeric/cache corruption.

Guidance: duplicate MIDDLE layers. The engine emits a boot-time advisory WARNING (llama-context.cpp, RYS engage block) whenever a plan duplicates a layer within the first/last max(3, n_transformer/8); it warns but does not block (the exact usable band is model-dependent). Correctness gate = coherence smoke via the CHAT endpoint (RYS intentionally changes the model, so logit-equivalence vs baseline is the wrong bar). Historical note: an earlier pass mis-labelled this an "SWA sub-cache engine bug" by testing only L0 (which is both SWA and first) on the noisy raw-greedy path — see [[project_rys_swa_dup_garbage]].

10b. RYS × MTP self-spec — the draft context must NOT inherit the plan (bug-2171, 0124)

RYS is a target-only capability. With RYS active and self-speculation on (--spec-type draft-mtp, Qwen NextN or the gemma4 assistant), the MTP draft llama_context is built from common_context_params_to_llama(params_base), so it inherited the target's params.repeat_layers — and its cparams.layer_plan got RYS-duplicated too. But the NextN head lives at the canonical il = n_transformer (n_layer − nextn_predict_layers), which the RYS eff→src map never contains, so draft-context init aborts with cpy_k: map_layer_ids MISS il=n_transformer → failed to create MTP context. Fix (0124, src/llama-context.cpp): the RYS plan-expansion block gates the spec on context type — rys_spec = (cparams.ctx_type == LLAMA_CONTEXT_TYPE_MTP) ? nullptr : params.repeat_layers (mirrored in sched_reserve()), so the draft runs the plain base stack while the target keeps RYS. This is lossless: greedy self-spec verifies every drafted token against the target, so the drafter's stack depth only moves acceptance/tps. In self-spec the NextN head consumes the target's already-RYS hidden state via shared memory — running the drafter plain is the intended composition, not a limitation.

11. References

• dnhkng, LLM Neuroanatomy II — the RYS-II study, Pareto table, pointer-based note ("The Models").
• dnhkng/RYS on GitHub — scanner, probes, model-builder config grammar.
• mergekit passthrough — the physical-merge convention (i,j) mirrors.