diff --git "a/docs/protocols/UPSTREAM_SYNC.md" "b/docs/protocols/UPSTREAM_SYNC.md" new file mode 100644--- /dev/null +++ "b/docs/protocols/UPSTREAM_SYNC.md" @@ -0,0 +1,1720 @@ +# Protocol — Upstream Sync + +opencoti is a soft fork of [opencode](https://opencode.ai). We sync +from upstream regularly. This document defines the rules that keep +syncs cheap. + +## Upstream + +- Tracked upstream: [mann1x/opencode](https://github.com/mann1x/opencode) + (dev fork; opencode-of-record is anomalyco/opencode). +- Default upstream branch: `dev`. +- Sync cadence: weekly is the target, more often when upstream is + hot. + +## The rule + +> **Every opencoti change is either an additive file outside the +> upstream tree, or a surgical hook inside it.** + +### Additive (preferred) + +A file that doesn't exist in upstream opencode. New files always live +in: + +- `docs/` +- `vendors/sources//`, `vendors/patches//` +- `packages/opencoti-*/` (new bun workspace packages) +- `opencoti/` (top-level non-bun roots, e.g. Go for `opencoti-server`) +- Top-level identity files we own: `README.md`, `CONTRIBUTING.md`, + `SECURITY.md`, `CLAUDE.md`, `STATE_SUMMARY.md`. + +Additive files never conflict on upstream merges. + +### Surgical hook + +A small edit to an existing upstream file. **Allowed only when +there is no additive way to achieve the effect.** + +A surgical hook MUST: + +1. Be the minimum-line edit that achieves the effect — one import + plus one call site, or one conditional. +2. Carry a marker comment on the same line or immediately above: + + ```ts + // opencoti-hook: — see docs/features/.md + ``` + +3. Defer to opencoti only when opencoti is enabled. With opencoti + disabled (or absent), the upstream behavior must be byte-identical. +4. Be listed in the **Registry** below. +5. Have an entry in the relevant child plan's "Milestones" or + "Surgical hooks" section. + +### What is NOT a surgical hook + +- A refactor of an upstream file "while we're here". File it + upstream instead. +- A rename of an upstream symbol. Wrap, don't rename. +- A behavior change that is unconditional. Gate it behind an + opencoti flag. + +## Sync workflow + +1. `git fetch upstream` +2. `git merge upstream/dev` (or rebase if the branch is local-only). +3. Resolve conflicts: + - In additive files: there should be no conflicts. If there are, + it means the file's path collided with new upstream content — + rename our file. + - In hook sites: keep the hook, accept upstream changes around + it. The marker comment is the anchor. +4. Run the conformance check: + `bun run scripts/check-hooks.ts` *(to be written — M0)* + It greps for `opencoti-hook:` markers and verifies they match the + Registry below. +5. Run typecheck and the test suites of every `packages/opencoti-*` + package. +6. Sync STATE_SUMMARY.md (note the upstream SHA we synced to). + +## Registry — known surgical hooks + +Add an entry in the same PR you add the hook. + +| Hook ID | File | Feature | Plan | +| --- | --- | --- | --- | +| `caliber-runtime` | `.gitignore` (trailing block) | Caliber pre-commit / session hooks runtime cache | This file (tooling, no per-feature plan) | +| `tiered-inference` (import) | `packages/opencode/src/session/llm.ts` (import block, just below `LLMRequestPrep` import) | F1 — Tiered Inference Engine | [features/tiered_inference.md](../features/tiered_inference.md) | +| `tiered-inference` (call site) | `packages/opencode/src/session/llm.ts` (inside `run` Effect.fn, just before the `flags.experimentalNativeLlm` seam) | F1 — Tiered Inference Engine | [features/tiered_inference.md](../features/tiered_inference.md) | +| `tiered-inference` (dep) | `packages/opencode/package.json` (`dependencies."@opencoti/tiers": "workspace:*"`) | F1 — Tiered Inference Engine | [features/tiered_inference.md](../features/tiered_inference.md) | +| `memory-tools-resolvability` (dep) | `packages/opencode/package.json` (`dependencies."@opencoti/memory": "workspace:*"`) | F4 M5 — Memory / Embedder | [features/memory_embedder.md](../features/memory_embedder.md) | +| `opencoti-default-plugins` (import) | `packages/opencode/src/config/config.ts` (just below the existing `ConfigPlugin` import) | F4 M5 follow-up — auto-wire opencoti plugins | [features/memory_embedder.md](../features/memory_embedder.md) | +| `opencoti-default-plugins` (call site) | `packages/opencode/src/config/config.ts` (in the config-loader Effect.fnUntraced body, just before the `return { config: result, … }`) | F4 M5 follow-up — auto-wire opencoti plugins | [features/memory_embedder.md](../features/memory_embedder.md) | +| `tiered-inference-setup` (import) | `packages/opencode/src/index.ts` (just after the existing `isRecord` import) | F1 M5.7 — Tier-0 setup CLI | [features/tiered_inference.md](../features/tiered_inference.md) | +| `tiered-inference-setup` (call site) | `packages/opencode/src/index.ts` (`.command(tierCommand)` after `.command(DbCommand)`) | F1 M5.7 — Tier-0 setup CLI | [features/tiered_inference.md](../features/tiered_inference.md) | +| `opencoti-server-autostart` (import) | `packages/opencode/src/config/config.ts` (just below the `opencoti-default-plugins` import) | F3 M3 — opencoti-server TS client + autostart | [features/opencoti_server.md](../features/opencoti_server.md) | +| `opencoti-server-autostart` (schema) | `packages/opencode/src/config/config.ts` (`opencoti: Schema.optional(Schema.Struct({ server: ... }))` block in the main config schema) | F3 M3 — opencoti-server TS client + autostart | [features/opencoti_server.md](../features/opencoti_server.md) | +| `opencoti-server-autostart` (call site) | `packages/opencode/src/config/config.ts` (in the config-loader Effect.fnUntraced body, just after the `opencoti-default-plugins` call site, before `return { config: result, … }`) | F3 M3 — opencoti-server TS client + autostart | [features/opencoti_server.md](../features/opencoti_server.md) | +| `opencoti-server-client` (dep) | `packages/opencode/package.json` (`dependencies."@opencoti/opencoti-server-client": "workspace:*"`) | F3 M3 — opencoti-server TS client + autostart | [features/opencoti_server.md](../features/opencoti_server.md) | + +The two TS edits carry `// opencoti-hook: tiered-inference — see docs/features/tiered_inference.md` markers; the `package.json` dep has no anchor comment (JSON has no comments) — this registry table is its only anchor. On upstream merges, if upstream renames any of these files or restructures the `run` Effect.fn around the call-site marker, the marker comments remain the canonical pointer for re-applying the hook. + +**F3 M4 (2026-05-23): no new registry rows.** M4 (session + +turn events flowing into opencoti-server) ships as a pure +plugin (`@opencoti/session-events-plugin`) that subscribes to +opencode's typed `Hooks.event` firehose. Since the plugin API +already exposes session.created/updated/deleted/error/idle as +typed events, no surgical hook into upstream source is +required — matching the M5-D2 precedent. Hook count stays +at 9 source markers + 4 JSON deps from F3 M3. See +`docs/features/opencoti_server.md` "M4" for the design-pivot +rationale. + +**F3 M5 (2026-05-23): no new registry rows.** M5 +(tier-engine event sink) ships as a plugin +(`@opencoti/tier-events-plugin`) that injects a `Telemetry` +implementation into `@opencoti/tiers` via a new module-level +slot (`registered-telemetry.ts`, mirroring the existing +`active-config.ts` overlay pattern). The slot is consulted +by `runtime.ts:99` after the explicit hook arg and +test-injected exec opts, before `noopTelemetry` — adding the +sink without touching upstream source. The `Telemetry` +interface itself was added during F1 M5 with the docstring +already saying F3 would implement it, so M5 lands in a seam +that was waiting for it. Hook count stays at 9 source +markers + 4 JSON deps from F3 M3 (unchanged through M4 and +M5). See `docs/features/opencoti_server.md` "M5" for the +seam design. + +**F3 M6 (2026-05-23): no new registry rows.** M6 (daemon +tool registry → opencoti sessions) ships as a plugin +(`@opencoti/server-tools-plugin`) that fetches the daemon's +`/v1/tools` catalog and injects a `Record` into +`@opencoti/tiers` via a new module-level slot +(`registered-server-tools.ts`, mirroring M5's +`registered-telemetry.ts`). The slot is consulted by +`runtime.ts`'s `mergeTools` call **last** — after the +synthetic-tier and memory-bridge tool records that were +already merged there since F1/F4 M5 — so the daemon tools +land in a seam that already existed for exactly this kind of +record-merge, no upstream source touched. Hook count stays at +9 source markers + 4 JSON deps from F3 M3 (unchanged through +M4, M5, and M6). See `docs/features/opencoti_server.md` "M6" +for the design-pivot rationale. + +**F3 M7 (2026-05-23): no new registry rows.** M7 (the pgvector +storage backend) is **entirely additive Go inside the +`opencoti/server/` root** — a new `internal/store/pgvector/` +package implementing the same `Store` interface family as +`internal/store/sqlitevec/`, plus a shared +`internal/store/storetest/` conformance harness and a pure-logic +`internal/store/common.go` both backends share. Backend selection +is a serve-time CLI flag (`--backend sqlite|pgvector`), not a build +flag, so no upstream file and no TS package is touched. Unlike +M4–M6, M7 ships **no plugin** — there is no opencode-side surface to +wire, since the backend is purely internal to the daemon. Hook count +stays at 9 source markers + 4 JSON deps from F3 M3 (unchanged through +M4, M5, M6, and M7). Note: M7 also moved the module + host Go +toolchain to 1.25 for the modern pgx/pgvector-go/testcontainers +releases — a build-environment change, not a sync-surface change. +See `docs/features/opencoti_server.md` "M7" for the backend design. + +**F3 M8 (2026-05-23): no new registry rows.** M8 (multi-session +coordination — presence, broadcasts, advisory locks) is **additive Go** +inside `opencoti/server/`: a new in-memory `internal/coord` hub plus an +`internal/server/coord.go` HTTP surface (including the daemon's first +SSE endpoint), injected as a concrete dependency on `server.Options`. +The new `@opencoti/coordination-plugin` is the ecosystem's first SSE +consumer, but it wires in like M4–M6: a string appended to +`OPENCOTI_DEFAULT_PLUGINS` in opencoti-owned `@opencoti/tiers`, consumed +by the already-registered `opencoti-default-plugins` call site — **not** +a new source marker. The count stays at 9 source markers + 4 JSON deps +from F3 M3 (unchanged through M4–M8). No upstream file and no store +backend touched. See `docs/features/opencoti_server.md` "M8". + +**F3 M9 (2026-05-23): no new registry rows.** M9 (opt-in shared +context) is **additive Go** inside `opencoti/server/`: a new +`internal/share` package (the share coordinator) plus an +`internal/server/share.go` HTTP surface, both reusing the existing +`Store` ACL methods (`SetSessionACL` / `ListCollections`) and the M8 +`coord.Hub` — no new `Store` method, no schema change, no store backend +touched. The `@opencoti/coordination-plugin` is **extended in place** to +consume the two new `collection.shared` / `collection.unshared` bus +events and to add an opt-in `share_session_collection` flag; it remains +auto-wired through the already-registered `opencoti-default-plugins` call +site, so no new source marker. The count stays at 9 source markers + 4 +JSON deps from F3 M3 (unchanged through M4–M9). See +`docs/features/opencoti_server.md` "M9". + +**F3 M10 (2026-05-24): no new registry rows.** M10 (the sqlite↔pgvector +`migrate` subcommand) is **additive Go** inside `opencoti/server/`: a new +`MigrationStore` optional interface on `internal/store` plus a per-backend +`migrate.go` implementing it and a new `cmd/opencoti-server/migrate.go` +subcommand. Like M7 it has **no opencode-side surface** — no plugin, no +HTTP endpoint, no TS client method — so there is nothing to auto-wire and +no source marker to add. It touches no upstream file, no schema (dedupe +reuses existing columns/indexes), and no `go.mod`/`go.sum` (deps already +present; the new code is stdlib `encoding/binary` + already-vendored +drivers). The count stays at 9 source markers + 4 JSON deps from F3 M3 +(unchanged through M4–M10). See `docs/features/opencoti_server.md` "M10". + +**F5 M0–M3 (2026-05-24 → 2026-05-29): no new opencode source markers.** +The Advanced KV Cache series (`0006`–`0030`) lives entirely in the +**vendored** `vendors/sources/llamafile/` tree, captured into numbered +patches under `vendors/patches/llamafile/`. The 6 surgical hooks added +during F4 M5+M3 (`tiered-inference` x3, `memory-tools-resolvability`, +`opencoti-default-plugins` x2 — see rows above) stay unchanged. F5 M3 +specifically also adds **two NEW non-upstream files inside the vendored +tree**: `vendors/sources/llamafile/llama.cpp/ggml/include/ggml-neo-pipeline.h` +and `.../ggml/src/ggml-neo-pipeline.cpp` (the NEO orchestrator). These +do **not** belong in this registry — they are not surgical hooks into +*upstream opencode*, they are additions to *vendored llama.cpp* (the +distinction in "What is NOT a surgical hook" above). They are tracked +in `vendors/patches/llamafile/README.md`'s 0030 row, which is the +canonical vendored-source registry. + +**F5-opt W1–W2 (2026-05-30): still no new opencode source markers.** The +concurrency=1 optimization round (`0036`, `0040`) likewise lives entirely +in vendored `llama.cpp`. `0036` (W1, PCIe probe) adds **NEW** non-upstream +`common/pcie-profile.{h,cpp}`. `0040` (W2, wholesale iqk CPU-FA import) adds +the **NEW** vendored `ggml/src/iqk/**` engine tree (29 files), plus +`ggml/include/ggml-iqk-flash-attn.h` + `ggml/src/ggml-iqk-flash-attn.cpp`, +and carries one `opencoti-hook: f5-opt-cpufa` marker in the *vendored* +`ggml/src/ggml-cpu/ops.cpp` dispatch site — which, being under +`vendors/sources/`, is **not** counted by the `packages/ docs/` grep and is +registered solely via the `0040` README row. The `opencoti-llamafile` +adapter wires `--iqk-flash-attn` through plain config (no marker). Marker +count in `packages/ docs/` is therefore unchanged: **20** per +`git grep "opencoti-hook:" packages/ docs/ | wc -l` (this counter has read +20 since F4 M5+M3; earlier "18/19" figures in this file are stale prose, +not a real source delta). See `docs/features/advanced_kv.md` for the +per-milestone history. + +**F5-opt W3–W4 + F5 M6 PolyKV/MTP (2026-05-30 → 2026-06-07): still no new opencode source +markers.** `0042` (W3, fused MoE up+gate), `0070` (W4, M7 Rolling KV), `0071` (cosmocc +state-IO, bug-269), `0072`/`0073` (M6 PolyKV — SharedKVPool + TurboQuant), and `0074` +(M6-S4 — speculative MTP draft head, #373/#404-410) all live entirely in vendored +`llama.cpp` (plus, for `0071`, the outer-vendor `llamafile/llamafile.{c,h}`, and for +`0073`, the outer-vendor `llamafile/build-functions.sh`) — **none** are hooks into upstream +*opencode* source, so none are counted by the `packages/ docs/` grep; each is registered +solely via its `README.md` row. `0074` touches only nested `llama.cpp/` files (24 files, +1 new: `src/models/gemma4-assistant.cpp`), no outer-vendor delta; its three surgical +upstream edits + the additive surface are detailed in the F5 M6-S4 block below. + +> **Update 2026-06-16 (llamafile 0.10.3 bump):** `0074` — and the gemma-specific `0076` / `0077` — are now +> **RETIRED**, superseded by upstream-native NextN/MTP + gemma4 E-series support in `dbe9c0c` (the +613-commit +> jump under the 0.10.3 bump). See the per-patch RETIRED banners below and +> `docs/protocols/LLAMAFILE_UPGRADE.md`. The active stack is **35 patches** +> (`0001-0007, 0010, 0020, 0030-0036, 0040, 0042, 0070-0073, 0075, 0078, 0079, 0080, 0081, 0082, 0083, 0084, 0085, 0086, 0087, 0088, 0089`); the three retired entries are +> kept below for audit history only (their patch files no longer exist). **Re: `0079` —** the gemma assistant +> drafter (retired `0074`) is **restored on 0.10.3 as `0079-gemma-assistant-mtp.patch`** to *coexist* with +> upstream-native NextN as `--spec-type draft-assistant` (#423, a deliberate two-engine union): native NextN +> does **not** cover Gemma-4 (iSWA, no NextN head), so the assistant drafter is Gemma's only MTP path. See the +> `opencoti F5 M6-S4 mtp (0.10.3 restore)` block near the end of this file. + +**`0070` marker count (S4 update, 2026-06-04):** the R2-a-era "6 markers" figure +is superseded. After the full Stage-3 position-window forward + the Stage-4 +`--kv-residency-mode` knob landed (and S4-2 **removed** the throwaway iSWA-spike ++ ~29 env-gated diagnostic scaffolds), `0070` now carries **~40** `opencoti-hook: +f5-rolling-kv` / `rolling-kv M7` markers across `ggml/include/{ggml.h,ggml-iqk-flash-attn.h}`, +`ggml/src/{ggml-backend.cpp}`, `ggml/src/ggml-cpu/{ggml-cpu.c,ggml-cpu.cpp,ops.cpp,ops.h}`, +`ggml/src/ggml-cuda/{fattn.cu,fattn.cuh,fattn-common.cuh,ggml-cuda.cu}`, +`ggml/src/iqk/iqk_flash_attn.cpp`, `src/{llama-context.cpp,llama-graph.cpp}` +(the canonical inventory is `git grep -n "opencoti-hook:" vendors/sources/llamafile/llama.cpp` += **54** total incl. the `f5-opt-fmoe`/`f5-opt-cpufa` markers from `0040`/`0042`). +The exact set is descriptive, not load-bearing: the `0070` README row is the +registry of record. **`0071`** carries no new in-tree marker — its 3-file edit +(the `llamafile_open_plain` def/decl + the `llama-mmap.cpp` COSMOCC fallback) is +registered via the `0071` README row. **Counter correction:** the +canonical opencode-source-hook figure is `git grep -h "opencoti-hook:" -- packages/` += **10**, unchanged by W3/W4. The *combined* `packages/ docs/` counter now reads +**22** (the W1–W2 note above cites 20); that +2 is **pure docs prose** — this +file's W2 paragraph and `advanced_kv.md` both now mention `f5-opt-cpufa` by name — +not new source hooks. That combined counter has always conflated real markers with +documentation mentions; treat `packages/`-only = **10** as canonical, and the +vendored-patch markers as registered by their `README.md` rows. + +**F5 M6 PolyKV S1 (2026-06-05): no new opencode source markers.** The +`opencoti-hook: poly-kv-pool` markers (`git grep -n "poly-kv-pool" +vendors/sources/llamafile/llama.cpp/tools/server` = **4**: 1 param block in +`server-task.h`, 2 parse lines in `server-task.cpp`, 1 gated prefill branch in +`server-context.cpp`) live entirely in vendored `llama.cpp` server source and are +additive + `shared_pool_slot < 0`-default-off (boot byte-identical). **Captured as +`vendors/patches/llamafile/0072-poly-kv-pool.patch` at M6-S3 (#372)** and registered +via that patch's `README.md` row — same mechanism as `0070`/`0071`. The +`packages/`-only canonical counter stays **10**. S1 runtime +prerequisite: the server must boot `--no-clear-idle` so the read-only pool slot's +KV stays device-resident for cross-seq `seq_cp` sharing (see `poly_kv.md` S1 / [[bug-367]]). + +**F5 M6 PolyKV S2 perf-path Level A (2026-06-06): GPU-resident turbo dequant-on-lift.** New +`opencoti-hook: turboquant perf-lift` markers — `git grep -n "turboquant perf-lift" +vendors/sources/llamafile/llama.cpp` = **10** across **6 files**: 3 in new additive opencoti files +(`ggml/src/ggml-cuda/turbo-cpy.cu`, `turbo-cpy.cuh`, `turbo-innerq-dev.cuh` — file headers) and 7 in +**4 edited** vendored files (`ggml-cuda/set-rows.cu` ×2 = include + accessor impl; `ggml-cuda/cpy.cu` +×3 = 2 includes + 1 dispatch branch; `ggml-cuda/ggml-cuda.cu` ×1 = CPY `supports_op` accepting +`turbo{2,3,4,8}_0→F16`; `src/llama-graph.cpp` ×1 = single-cast `turbo→F16` lift). All additive + +default-inert (the CPY branch only fires for a `turbo→f16` cast, which only exists when `-ctk/-ctv +turboN` is set; f16/q8/q4 paths untouched, boot byte-identical). **Captured in +`vendors/patches/llamafile/0073-turboquant-kv.patch` at M6-S3 (#372)** and registered via that patch's +`README.md` row — same mechanism as `0070`/poly-kv-pool. The +`packages/`-only canonical counter stays **10**. See `poly_kv.md` S2 perf-path Level A. + +**F5 M6 PolyKV S2 perf-path Level B (2026-06-06): fused FA-VEC turbo + new `GGML_OP_TURBO_WHT` op.** +New `opencoti-hook: turboquant perf-lift (… Level B)` markers, additive on top of Level A. **New ggml op +`GGML_OP_TURBO_WHT`** appended at the end of the op enum (after `FLASH_ATTN_TAIL_PARTIAL`, before +`GGML_OP_COUNT`) — same append-at-end discipline as `MOE_FUSED_UP_GATE`/`STREAMING_FLASH_ATTN`; the +host/DSO `GGML_OP_COUNT` (now **100**) and the name/symbol arrays + both static_asserts in `ggml.c` move +in lockstep, CPU forward is an abort-stub (CUDA-only op, like `STREAMING_FLASH_ATTN`). New additive files: +`ggml-cuda/turbo-wht.{cu,cuh}` + `turbo-wht-core.cuh`, `ggml-cuda/fattn-turbo.cuh`, +`template-instances/fattn-vec-instance-turbo{2,3,4}_0-*.cu` (3, generator-emitted). Edited vendored files +(surgical, default-inert — only fire when `-ctk/-ctv turbo{2,3,4}` is set): `ggml.h`+`ggml.c` (op enum + +`ggml_turbo_wht` ctor), `ggml-cpu/ggml-cpu.c` (abort case + planner n_tasks=1), `ggml-cuda/ggml-cuda.cu` +(TURBO_WHT dispatch + supports_op), `ggml-cuda/fattn-common.cuh` (turbo `get_vec_dot_KQ`/`get_dequantize_V` +cases), `ggml-cuda/fattn-vec.cuh` (`is_turbo_K` config), `ggml-cuda/fattn.cu` (turbo→VEC gate), +`template-instances/generate_cu_files.py` (turbo instance loop), `src/llama-graph.cpp` (`build_attn_mha` +fused gate + prefill-hybrid `TBV_FUSE_MAX_NQ`). f16/q8/q4 paths untouched, boot byte-identical. **Captured +together with Level A in `0073-turboquant-kv` at M6-S3 (#372).** See `poly_kv.md` S2 perf-path Level B. + +**F5 M6 PolyKV patch numbering (S3, #372): `0072`/`0073`, NOT the plans' stale `0060`/`0061`.** The +build pipeline applies patches in **pure lexical sort** order (`build-pipeline.ts`, `patches.sort()`); +there is no non-lexical override. The TurboQuant FA-VEC edits (`fattn.cu`, `llama-graph.cpp`, +`ggml-cuda.cu`) sit **on top of** `0070`'s rewrite of those same files, so they MUST sort **after** +`0071` — `0060`/`0061` would sort before `0070` and break the chain (M7 shipped before M6, so the +original plan labels predate it). `0073` also carries the outer-vendor `llamafile/build-functions.sh` +host-build source-list add for the new `ggml-turbo-quant.c` TU. Both patches reverse-apply clean against +live and forward-reproduce the live tree byte-for-byte (snapshot-diff capture, same as `0070`/`0071`). +S0 asymmetric `-ctk q8_0 -ctv q4_0` needs **no patch** (rides M7 dequant-on-lift natively). + +**F5 M6-S4 P2 (#406, 2026-06-07): Gemma4 MTP speculative-decode graph + inert (verify-only) +scheduler.** Ported from the AtomicBot `atomic-llama-cpp-turboquant` fork. All new code carries the +`// opencoti F5 M6-S4 mtp` marker (same marker family as P1). **Default-inert:** nothing in the normal +decode/server path invokes the MTP machinery — `llama_decode_mtp` is the only entry and it is called by +nobody in-tree at P2 (active drafter is P3/P4). The patch is **additive-dominant** with **three surgical +edits to existing upstream functions** that must be re-applied on upstream sync (the marker comment is the +real anchor; line numbers below are best-effort): + +- **`src/llama-graph.h` — `llm_graph_params::allow_reuse` `token_compat` relaxation** (~line 572). The + inlined ubatch token/embd compatibility expression is replaced with an MTP-aware named `token_compat` + bool: MTP graphs feed BOTH token and embd, so steps 1..N-1 of one `decode_mtp` can reuse the encoded + graph. Default (non-MTP) branch is byte-identical to upstream. +- **`src/llama-graph.cpp` — `llm_graph_result::reset()` nulls `t_argmax`** (~line 805). One added line next + to the other `t_* = nullptr;` lines so the new MTP-only `t_argmax` field is reset like its siblings. +- **`src/llama-model.cpp` — `build_graph` `case LLM_ARCH_GEMMA4` MTP sub-branch** (~line 8723). The target + arch case gains a `params.gtype == LLM_GRAPH_TYPE_MTP` branch that builds `llm_build_gemma4_mtp` from the + nested `mtp_assistant`; the `else` (iswa) path is unchanged. The `LLM_ARCH_GEMMA4_ASSISTANT` case keeps + its P1 throw-stub. + +Everything else is **purely additive** (new functions/fields/file, no upstream-line mutation, marker-tagged): +`LLM_GRAPH_TYPE_MTP` enum value; `llm_graph_input_mtp` class (+`set_input`/`can_reuse`); `t_argmax` +field + `get_argmax()` accessor; `llm_graph_context::build_attn_mtp`; the **new file** +`src/models/gemma4-assistant.cpp` (+ its source-list entry in **both** `llama.cpp/BUILD.mk` — +the authoritative cosmocc build list, the one the host binary actually links — and `src/CMakeLists.txt` +as a mirror; + `llm_build_gemma4_mtp` decl in `src/models/models.h`); `llama_kv_cache::mtp_slot_info` + `llama_kv_cache_iswa::init_mtp`; the synchronous +`llama_context` MTP machinery (`graph_params_mtp`, `ensure_sched_mtp`, `process_ubatch_mtp`, +`graph_compute_mtp`, `decode_mtp` facade → `decode_mtp_sync`) + members `sched_mtp`/`gf_res_prev_mtp`; the +C API `llama_decode_mtp` (+ `include/llama.h` decl); and a `#include "llama-kv-cache-iswa.h"` in +`llama-context.cpp`. **Adaptations vs fork** (documented at the call sites): `build_attn_mtp` passes +`ggml_turbo_wht(..., /*scale=*/nullptr)` instead of the absent `get_turbo_innerq_scale_inv()` (OURS folds +the scale into the device-resident kernel); the `decode_mtp` facade is rerouted unconditionally to +`decode_mtp_sync` (async worker is P4); `ensure_sched_mtp`/`graph_compute_mtp` drop the async-only +`mtp_worker` spawn and `backend_cfg_mu` lock (sync-only P2 path). **Known risk R1 (double-Q-rotation):** if +the MTP cross-read target K/V enter `build_attn_mha` as turbo (fused branch), Q is rotated twice — graph +still *builds* (P2 inert), correctness is a P3 gate. + +> **⚠️ RETIRED 2026-06-16 — llamafile 0.10.3 bump (#464, B2).** `0074-speculative-mtp.patch` is **dropped from +> the rebased stack.** The nested llama.cpp base advanced `5e9c63546`→`dbe9c0c` (+613 commits; pin +> `vendors/pin/llamafile.txt` → `7fca8b2 0.10.3`), which ships **upstream-native NextN/MTP**: `255582687` +> "llama + spec: MTP Support (#22673)" (NextN self-attention MTP graph + the `n_rs_seq` recurrent rollback ring, +> a proven ancestor of `dbe9c0c`). The hand-wired `gemma4_assistant` draft head + its `tok_embd`-on-OUTPUT-buft +> placement fix are superseded *for throughput*; **qwen** MTP rebases onto upstream-native `--spec-type +> draft-mtp`, but the **gemma** assistant drafter — which native NextN does **not** cover (Gemma-4 is iSWA, +> no NextN head) — is **restored on 0.10.3 as `0079-gemma-assistant-mtp.patch`**, coexisting as `--spec-type +> draft-assistant` (#423). The 0.10.1-base `0074` *diff* is dropped (it did not apply across the +613-commit +> jump); the *feature* lives on in `0079` — see the active block near the end of this file. The block below is retained for audit history; the patch file no longer +> exists. See `docs/protocols/LLAMAFILE_UPGRADE.md`. + +**F5 M6-S4 P3/P4 (#407-410, 2026-06-07): active drafter + throughput win — CAPTURED as `0074-speculative-mtp.patch`.** +P3 activated the synchronous single-token MTP forward (`decode_mtp` → `decode_mtp_sync`, +`process_ubatch_mtp`, the C API) and proved it **logit-equivalent** to plain greedy +(`.opencoti/s4-p3-logit-equiv.sh`: `real_frac=0.0`, 100% draft acceptance). P4 made it a measured +**decode-throughput win**: the assistant's weight-tied dense LM head (`tok_embd`, ~277 MiB) is +classified as an INPUT tensor and was hard-pinned to the CPU buft (`llama-model.cpp:~2716` +`dev_input`), so the per-draft-step full-vocab (262144) output mul_mat ran as a **CPU island** that +stalled the GPU (~95% idle, `GGML_SCHED_DEBUG=2`). The fix (one surgical edit, marker-tagged) routes +the `gemma4_assistant` `tok_embd` through the **OUTPUT** buft (`dev_output`, GPU when the output layer +is offloaded) instead of `dev_input` → MTP-on **114.0 tok/s vs off 78.3 = 1.46×** (seq 113.5; +seq==fused byte-identical; acceptance 0.769/0.606). `0074` reverse-applies clean against live and +forward-reproduces the live tree byte-for-byte (snapshot-diff capture from base `0001-0042 + 0070 + +0071 + 0072 + 0073`, same mechanism as `0070`-`0073`; verified both directions, 24 files, 0 diffs). +TS adapter knobs: `mtpHead`/`specType`/`draftNGpuLayers` (`--mtp-head`/`--spec-type mtp`/`-ngld`) in +`packages/opencoti-llamafile/src/{config,launch}.ts`. The reverted sparse-argmax centroid fast path +(dead for this dense GGUF) is banked at `.opencoti/banked/mtp-sparse-argmax.md`, not shipped. + +**F5 M6-S2 #411 (2026-06-07): D=512 fused FA-VEC for turbo2/turbo3 — CAPTURED as `0075-d512-turbo-vec.patch`.** +`0073`'s Level-B fused turbo read covered head_dim∈{128,256}; the Gemma-4 A4B GLOBAL layers use +head_dim 512 (`key_length=512`) and therefore fell back to the Level-A turbo→f16 materialize (a decode +tax that scales with the global-cache `n_kv`). `0075` adds the D=512 fused FA-VEC instances for +turbo2/turbo3 (`DECL_FATTN_VEC_CASE(512, …)` in `fattn-vec.cuh` + the two template-instance `.cu` + +`generate_cu_files.py` autogen line; `fattn.cu` kernel-selection relax + dispatch; `llama-graph.cpp` +`build_attn_mha` `fused_turbo_512` gate). The VEC kernel is already **D-generic** (`nthreads`/ +`nthreads_V` derive from D arithmetically; at D=512/ncols=1 the KQ smem is ~4–8 KB and VKQ ~16 regs — +well within the 48 KB static-smem cap, verified compile + run clean). **No new surgical hook** — every +edit sits inside `0073`'s existing turbo regions; **no new ABI; no new flag** (the D=512 fused path +auto-engages for turbo2/3 decode via the existing `TBV_FUSE_MAX_NQ` knob, default 8). Correctness is +**Parseval-EXACT** (the WHT is block-diagonal over 128-blocks, so D=512 = 4 Parseval-exact 128-blocks): +fused-vs-materialize logit-equivalence `real_frac=0.0` for BOTH turbo3 and turbo2 (`.opencoti/p411-d512-gate.sh`). +Perf — turbo3 @ ctx 27.7k, fused **41.0 vs materialize 29.7 tok/s = +38%**, scaling with `n_kv` +(`.opencoti/p411-perf.sh`). Snapshot-diff capture (6 files, 95 lines, 0 new) from base +`0001-0042 + 0070 + 0071 + 0072 + 0073 + 0074`; verified both directions, 0 diffs. **No TS adapter +change** (no user-facing flag). + +When you add a hook, append a row above, replacing the placeholder +with: stable hook id, file path + best-effort line number (line moves +on upstream syncs are normal — the marker comment is the real +anchor). + +## Why we're strict + +Every undiscoverable edit to an upstream file is a future merge +conflict that someone (probably us) will resolve at 2 AM during a +release crunch. Strictness up front is the cheapest tax. + +### `opencoti F5 gemma4-shared-kv` (F5 — elastic E-series KV-layer-sharing) — ⚠️ RETIRED 2026-06-16 + +> **RETIRED at the llamafile 0.10.3 bump (#464, B2).** `dbe9c0c` loads gemma4 +> `gemma4.attention.shared_kv_layers` natively (its loader no longer requires `attn_k`/`attn_k_norm` on every +> layer), so the `TENSOR_NOT_REQUIRED` loader gate described below is **superseded by upstream native** support +> and was dropped from the rebased stack. `0076-gemma4-shared-kv.patch` no longer exists; entry kept for audit +> history. See `docs/protocols/LLAMAFILE_UPGRADE.md`. + +One *vendored-source* marker (`// opencoti F5 gemma4-shared-kv (#417)`) in +`vendors/sources/llamafile/llama.cpp/src/llama-model.cpp`, +in the `LLM_ARCH_GEMMA4` `create_tensor` loop. The elastic Gemma-4 E-series (E2B/E4B) declare +`gemma4.attention.shared_kv_layers` (E2B 20/35, E4B 18/42): the trailing layers carry no +`attn_k`/`attn_k_norm` and reuse an earlier layer's KV cache. The graph side +(`src/models/gemma4-iswa.cpp` `hparams.has_kv(il)` guard), `hparams.has_kv()` +(`src/llama-hparams.cpp`), and the KV-cache layer-reuse callback (`llama-model.cpp`, the +`arch == LLM_ARCH_GEMMA4` `layer_reuse_cb`) were already brought over by the MTP/TurboQuant +port; the only missing piece was the loader requiring `attn_k`/`attn_k_norm` on every layer. +The hook gates both with `const int kv_flags = hparams.has_kv(i) ? 0 : TENSOR_NOT_REQUIRED;`. +Inert for dense Gemma-4 (A4B/12B/31B): they set `n_layer_kv_from_start = n_layer`, so +`has_kv(i)` is true for all `i` and `kv_flags ≡ 0` — the `create_tensor` calls are bit-identical +to upstream. **Captured as `0076-gemma4-shared-kv.patch`** (snapshot-diff vs the 0075-applied +base; reverse-applies clean + forward-reproduces live byte-for-byte). Unblocks E2B/E4B load +(was bug-438). Marker convention: this is a **vendored-source** `// opencoti ` marker +(patches/README §"In-tree markers vs surgical hooks") — NOT an `// opencoti-hook:` marker, so the +`git grep "opencoti-hook:" packages/ docs/` inventory is unchanged. + +### `opencoti F5 M6-S4 mtp ordered-gpu` (F5 M6-S4 — ordered-embeddings draft-head GPU placement) — ⚠️ RETIRED 2026-06-16 + +> **RETIRED at the llamafile 0.10.3 bump (#464, B2).** This gemma4-MTP ordered-drafter GPU-placement hook is a +> follow-on to `0074` and is superseded by the same **upstream-native NextN/MTP** convergence (`255582687`, +> #22673) under `dbe9c0c`. `0077-gemma4-mtp-ordered-gpu.patch` was dropped from the rebased stack; entry kept +> for audit history. See `docs/protocols/LLAMAFILE_UPGRADE.md`. + +One *vendored-source* marker (`// opencoti F5 M6-S4 mtp ordered-gpu (#418)`) in +`vendors/sources/llamafile/llama.cpp/src/llama-model.cpp`, in the +`LLM_ARCH_GEMMA4_ASSISTANT` `create_tensor` path inside `if (hparams.use_ordered_embeddings)`. +The follow-on to 0074's dense `tok_embd` fix: the ordered ("efficient") draft head reads +`mtp_centroids` (centroid mul_mat) and `mtp_token_ordering` (get_rows) every draft step, and +llama.cpp default-classifies both as inputs pinned to the CPU buft (`dev_input`) — a CPU island +that fragments the MTP graph (`GGML_SCHED_DEBUG` splits=2, assistant `CPU_Mapped` 277 MiB) and +makes MTP-on *slower* than off on the ordered drafters (E2B/E4B). The hook routes both through the +OUTPUT buft (`pimpl->dev_output.buft_list`, GPU when the output layer is offloaded) via the +`ml.create_tensor(hparams, &pimpl->cpu_buft_list, dev_output…)` form — identical mechanism to 0074. +Result: assistant `CPU_Mapped` 277→0.80 MiB, draft `dur(g)` 783→300 ms; E2B 0.87×→1.22×, +E4B 1.34×→1.42× (RULER 32k, q8_0 KV; acceptance bit-identical pre/post). **Inert for the dense +drafters** (A4B/12B/31B): they set `use_ordered_embeddings=false` and never enter this branch. +**Captured as `0077-gemma4-mtp-ordered-gpu.patch`** (snapshot-diff vs the 0076-applied base; +reverse+forward byte-identical). Vendored-source marker — not counted by the `opencoti-hook:` +inventory. + +### `opencoti F5 dca` (F5 — Dual Chunk Attention, training-free long-context) + +The DCA marker family `// opencoti F5 dca` (187 occurrences) spans the vendored llama.cpp tree — +this is a substantial additive feature, not a one-line hook. Surface: + +- **New files** `vendors/sources/llamafile/llama.cpp/src/dca.{cpp,h}` — the three-regime builder + `build_attn_dca` / `build_attn_dca_core` / `build_attn_inp_dca` and the + `ggml_flash_attn_ext_dca_fused` host wrapper (INTRA `pos%c` / SUCC `+c` / INTER `2c`, merged by + online-softmax). +- **KV-cache fill** `src/llama-kv-cache.cpp` `set_input_dca` — #635 INTER-Q = `2c` dense + position-based fill + the #617B sticky `is_fragmented` fragmented-mask fallback (masks left NULL on + the contiguous fast path). +- **Per-arch dispatch hooks** in `src/models/{gemma4-iswa,qwen2,qwen3,qwen35,qwen35moe,qwen3moe}.cpp` + (+ `models.h`, `llama-arch.cpp`, `llama-graph.{h,cpp}`, `llama-kv-cache.h`, `llama-kv-cells.h`, + `llama-context.cpp`) — gemma4 hd512 (global layers), qwen2/3 hd128, qwen3.5 family hd256, incl. the + qwen35 `nextn_predict_layers` MTP-block skip (#622). +- **Runtime knobs** `src/llama-cparams.h` (`dca_enabled` / `dca_chunk_size` / `dca_yarn_factor`) + `common/{arg,common}.{h,cpp}` + args `--dca` / `--dca-chunk-size` / `--dca-yarn-factor` (env `LLAMA_ARG_DCA*`); `tools/server/server-context.cpp`. + `--dca-yarn-factor` is **consumed** in `build_dca_rope` (`dca.cpp`) as the YaRN mscale — + `dca_attn_factor = attn_factor * cparams.dca_yarn_factor` into `ggml_rope_multi`/`ggml_rope_ext` + (marker `// opencoti-hook: dca-yarn-factor (bug-740 / #550)`, patch `0087`; default 1.0 ⇒ inert/byte-identical). + **Characterized as no-win for retrieval** (over-sharpens; default 1.0 correct — see README `0087` row). +- **New ggml op** `GGML_OP_FLASH_ATTN_EXT_LSE` (`ggml/include/ggml.h` + `ggml.c`, `ggml-backend.cpp`, + `ggml-cpu/*`, and the fused MMA kernel + LSE combine in `ggml/src/ggml-cuda/{fattn*,ggml-cuda.cu}`): + flash-attention emitting a NORMALIZED O packed with per-row lse, so the three regimes merge exactly. + +Default-OFF (`cparams.dca_enabled=false`) is byte-identical to pre-0078 (the contiguous fast path leaves +the fragmented masks NULL). Constraints today: `head_dim` in {128,256,512}, `DK==DV`, **f16 KV required** +(`dca.cpp` assert — to be generalized to all KV types, incl. bf16 / scalar / turbo / TCQ, in the follow-on +all-KV-types milestone). **Captured as `0078-dca.patch`** (clean-room capture from the sibling +`opencoti-dca` repo, 37 files / 2 new; landed byte-identical, `git apply --check` clean on the +0077-applied tree, production-clean of `DCA617*` debug). Marker convention: these are **vendored-source** +`// opencoti ` markers (patches/README §"In-tree markers vs surgical hooks") — there are **zero** +`// opencoti-hook:` markers in 0078, so the `git grep "opencoti-hook:" packages/ docs/` inventory is +unchanged. **ABI:** adds the `GGML_OP_FLASH_ATTN_EXT_LSE` ggml op — the cosmocc binary + `ggml-cuda.so` +DSO must be rebuilt and paired. + +### `opencoti F5 M6-S4 mtp (0.10.3 restore)` (F5 M6-S4 — Gemma-4 assistant-drafter MTP, restored onto 0.10.3) — `0079` + +The gemma assistant drafter retired as `0074` at the 0.10.3 bump (its 0.10.1-base snapshot-diff did not apply +across the +613-commit jump to `dbe9c0c`) is **restored on the new base as `0079-gemma-assistant-mtp.patch`** +(#423), re-exposing it as a first-class `--spec-type draft-assistant` that **coexists** with upstream-native +NextN MTP (`--spec-type draft-mtp`). This is a deliberate **two-engine union**, not a reversal of the +retirement: native NextN covers the qwen-family NextN archs; the assistant drafter covers **Gemma-4**, which +has **no native NextN head** (iSWA) — so for Gemma it is the *only* MTP path. The user mandate (#423) is +explicit: do not drop ours; extend/restore it so it works alongside every other development (PolyKV, +rolling-KV, turbo, DCA). + +**No enum/graph collision:** `COMMON_SPECULATIVE_TYPE_MTP` (ours) vs `COMMON_SPECULATIVE_TYPE_DRAFT_MTP` +(native); `LLM_GRAPH_TYPE_MTP` (ours) vs `LLM_GRAPH_TYPE_DECODER_MTP` (native). The ~0.4B Q-only +cross-attention drafter (`LLM_ARCH_GEMMA4_ASSISTANT`) loads **into** the gemma4 target via +`llama_model_load_mtp_from_file` (no second `llama_context`, no second KV cache) and reads the target KV +read-only at decode (`build_attn_mtp`). It rides the 0.10.3 **seq-aware speculative framework** +(`common_speculative_process`/`_draft`/`_accept`/`_need_embd` public hooks, which replace the old server +`h_idx`/`seq_id`/`set_h_idx` plumbing) — which is *why* it composes with PolyKV / multi-seq for free. + +All new code carries the `// opencoti F5 M6-S4 mtp` marker (vendored-source, **not** `opencoti-hook:` — +registered via the `0079` README row + this block; the `packages/ docs/` counter is unchanged). The patch is +**additive-dominant**; the surgical edits to existing 0.10.3 upstream functions that must be re-applied on the +next upstream sync (the marker comment is the real anchor; line numbers best-effort): + +- **`src/llama-graph.cpp` — `llm_graph_input_attn_kv_iswa::set_input` buffer-guard decouple** (**bug-508**). + 0.10.3 nests the `set_input_kq_mask` call **inside** `if (self_k_idxs && self_k_idxs->buffer)`. The + read-only MTP draft graph (`build_attn_mtp`) prunes the KV-write idxs (their galloc buffer is null) yet still + reads the mask — so the nested guard would skip the *live* mask and feed garbage (a silent re-introduction of + bug-404). Fix: **decouple** — each `self_{k,v}_idxs` / `self_kq_mask{,_swa}` / rot setter now guards on **its + own** `->buffer`. **Byte-identical for every non-MTP graph** (all buffers live ⇒ every branch taken, as + upstream). +- **`tools/server/server-context.cpp` — leading `has_dft() && draft-assistant` branch.** Before the existing + `has_dft()` draft-model path (now `else if`), a new branch detects `COMMON_SPECULATIVE_TYPE_MTP` in + `--spec-type`, calls `llama_model_load_mtp_from_file(model_tgt, params_dft.model.path, mparams_dft)`, and + leaves `model_dft`/`ctx_dft` **null** (null-safe: every `ctx_dft` checkpoint path early-returns on a null + context). +- **per-arch loader + graph-dispatch plumbing** — a per-model class `llama_model_gemma4_assistant` registered + via `llama_model_mapping` (the 0.10.1 giant-switch loader is gone) and a `params.gtype == LLM_GRAPH_TYPE_MTP` + sub-branch routing to `llm_build_gemma4_mtp` (new file `src/models/gemma4-assistant.cpp`); the base gemma4 + iSWA path is unchanged. + +Everything else is additive (the `LLM_GRAPH_TYPE_MTP` enum, `llm_graph_input_mtp` class + +`set_input`/`can_reuse`, `build_attn_mtp`, the `mtp_slot_info`/`init_mtp` KV machinery, the C API +`llama_decode_mtp`, and the `--spec-type draft-assistant` arg). **Default-inert:** the MTP path is reached only +with the assistant loaded + `--spec-type draft-assistant`; plain decode and native `draft-mtp` are unchanged. + +**Touches zero `.cu`/`.cuh`** — the assistant rides the existing fattn/turbo/DCA CUDA kernels, so landing it on +the live tree is a **cosmocc host rebuild only** (no DSO rebuild/restamp). Capture: snapshot-diff from the spike +`0078`-tip base (23 files, 1 new `src/models/gemma4-assistant.cpp`, 0 index lines); **reverse + forward +byte-identical** in the spike; spike **host build GREEN** (compiles + links; `strings` shows `draft-mtp` and +`draft-assistant` coexisting). **Live-binary composition gates are PENDING** (#461): assistant-MTP × {PolyKV / +POSITION_WINDOW / turbo / DCA} logit-equivalence (`real_frac=0`, the locked "non-MTP works in MTP" gate) runs on +the live CUDA binary, gated on a host rebuild + GPU availability. **Applies after `0078`.** + +#### Upstream-alignment overlay (#476 — adopt upstream `#23398`/`#24282` DATA-plane, keep our control-plane) + +Upstream llama.cpp merged Gemma-4 MTP (**PR #23398** = release **b9549**, + E-series **#24282**). `0079` now +carries upstream's **data-plane identity** so a future llamafile bump onto a base containing #23398 reconciles +the gemma assistant-MTP as a clean *delete of the identity layer*, keeping only our control-plane delta. The +rename folded into `0079` (S1; re-runs of the same hunks, no new files): + +| surface | was (ours) | now (upstream-aligned) | +|---|---|---| +| arch string | `gemma4_assistant` | **`gemma4-assistant`** (hyphen) | +| proj tensors | `mtp.pre_projection` / `mtp.post_projection` | **`nextn.pre_projection`** / **`nextn.post_projection`** (`LLM_TENSOR_NEXTN_PROJ_{PRE,POST}`) | +| E-series tensors | `mtp.centroids` / `mtp.token_ordering.weight` | **`masked_embd_centroids`** / **`masked_embd_ordering`** (bare, NO `.weight`) (`LLM_TENSOR_MASKED_EMBD_*`) | +| backbone-width KV | `gemma4_assistant.n_embd_backbone` | **`%s.embedding_length_out`** (`LLM_KV_EMBEDDING_LENGTH_OUT`); hparams `n_embd_out_impl` | +| layers KV | (derived) | adds **`%s.nextn_predict_layers`** (= block_count) | + +**What we deliberately do NOT adopt — `ctx_other`.** Upstream's assistant mechanism is a **second `llama_context`** +that aliases the target KV by struct-copying its layer record + mirroring cell metadata each step +(`llama-kv-cache.cpp` `apply_ubatch: if (other) { v_cells = other->v_cells; return; }`; `seq_*`/`update` all +`if (other) return;`). It assumes the target KV is **vanilla: single-owner, contiguous, unquantized** — the +opposite of opencoti's KV moat, and it is *independently broken* under quantized KV (public PR: 0% acceptance +with `-ctk q8_0 -ctv q8_0`). It would conflict with **all five** gate-required features (PolyKV per-stream +layers, rolling-KV POSITION_WINDOW, M2/M3 head-split, TurboQuant KV, DCA). So we **keep the opencoti +control-plane**: single-context `build_attn_mtp` read-only cross-read (`ctx_dft == nullptr`, load INTO target via +`llama_model_load_mtp_from_file`); ordered/centroid masked-embedder head; `tok_embd`→OUTPUT buft; fused N-step; +`--spec-type draft-assistant` (CLI string, not GGUF identity — preserves the 2-engine dispatch vs qwen +`draft-mtp`). Bespoke keys with no upstream equivalent (`n_centroids`, `centroid_top_k`, `use_ordered_embeddings`, +`attention.k_eq_v`, `requires_target_arch`) remain opencoti-private under the `gemma4-assistant.*` prefix — +upstream loaders ignore unknown keys. + +**The rename is semantically INERT** — proven at three levels: (1) the re-formatted A4B drafter GGUF +(`.opencoti/gguf-upstream-rename.py`, Q8_0 bytes copied verbatim) has whole-file tensor-data sha256 == source; +(2) live boot-smoke on the renamed CUDA host binary loads `gemma4-assistant`, routes `draft-assistant`, drafts +(44% accept), decodes coherently; (3) **`real_frac = 0` logit-equivalence** on the renamed engine +(`.opencoti/s4-p3-logit-equiv.sh`, A4B v6-coder Q4_K_M). The `0079` patch is **surgically confined**: 16 of its +23 file-stanzas are **byte-identical** to the pre-rename `0079`; only the 7-file data-plane surface +(`llama-arch.{cpp,h}`, `llama-context.cpp`, `llama-hparams.h`, `llama.cpp`, `common/speculative.cpp`, +`src/models/gemma4-assistant.cpp`) changed — the entire control-plane (kv-cache, graph, model registration, +server branch, base gemma4) is untouched. **Chain proof (G33/G35):** `0079` regenerated as +`diff(clean-0078-replay, live)` where the replay is a fresh `git archive HEAD` + llamafile `apply-patches.sh` +overlay + strict `git apply` of `0001..0078`; the FULL chain `0001..0079` then strict-applies from pristine +(**0 rejects**), final tree **== live byte-identical across 2941 source files**, reverse round-trips. **A4B +beachhead only** — re-formatting the other 4 drafters (E2B/E4B/12B/31B) + the full composition matrix on the +renamed engine are FOLLOW-ON (#461/#476). Touches zero `.cu`/`.cuh` (host rebuild only). + +**`#408` OUTPUT-buft placement re-claimed on 0.10.3 (#484).** The `tok_embd`→OUTPUT-buft win listed in the kept +control-plane is not free on the bumped base: 0.10.3's `ml.create_tensor` selects the device buft by the tensor's +`llm_tensor_info` layer, and `TOKEN_EMBD` classifies `LLM_TENSOR_LAYER_INPUT` → **CPU** buft, so the per-model +loader fragments the assistant's per-step full-vocab matmul into a CPU island (MTP-on *slower* than off at short +ctx; bug-540). Re-claimed with an **additive** `llama_model_base::create_tensor_output_head(...)` (+ml-less overload, +`src/llama-model.{cpp,h}`) that passes `dev_output.buft_list` in the INPUT slot to force GPU OUTPUT placement, with +**normal** load accounting (NOT `TENSOR_DUPLICATED` — `tok_embd` is the sole-use tied output head). Gated: +`real_frac = 0` (placement is byte-equivalent) **+ 1.57× at RULER-32k** (off 56.6 → on 88.8 tok/s, 87% accept) — +exceeds the historical 1.46×. Folded into `0079` (still 23 files; the 3 stanzas `llama-model.{cpp,h}` + +`gemma4-assistant.cpp` grew, full `0001..0079` chain re-proven from pristine). + +**PolyKV multi-slot support folded into `0079` (#485).** Two surgical hooks in +`tools/server/server-context.cpp` (both `// opencoti F5 M6-S4 mtp (#485)`) let the Gemma assistant +drafter compose with the SharedKVPool multi-tenant path: (1) at the spec-fit pre-measure (~:809), +`spec_assistant_fit` skips the standalone draft-`llama_context` memory probe for the assistant engine +(`gemma4-assistant` legitimately refuses to be a primary `-m` model, so the probe would log a benign +`failed to measure draft model memory` warning — the source of the earlier bug-550 misdiagnosis); (2) at +the assistant-load branch (~:940), a boot guard requires `--kv-unified` when `--parallel >1`, because the +single-context cross-read (`build_attn_mtp` → `build_attn_mha`) infers the stream count from the target +K's 4th axis (`n_stream = k->ne[3]`, `src/llama-graph.cpp:2115`) and the per-stream-split KV layout +(`0031`) hands back a multi-stream tensor the single-token draft query cannot be partitioned against +(`ggml.c:3754` reshape assert). PolyKV forces `--kv-unified`, so it composes (`real_frac=0` + ~93% draft +acceptance @ `--parallel 2 --kv-unified`); the guard turns the unsupported non-unified multi-slot config +into a clean boot error instead of a mid-request SIGABRT. Within the existing `0079` 23-file surface (the +two hooks live in the already-touched server stanza); full `0001..0079` chain re-proven byte-identical +from pristine. Non-unified multi-slot (an `n_seq`-aware assistant draft) is tracked under #469. + +### `0080-mtp-d512-fa` — D512/gqa≤2 flash-attn for the Gemma-4 assistant-MTP drafter (#491+#493, bug-567) + +The Gemma-4 E4B assistant-MTP drafter's single GLOBAL layer is head_count 4 / head_count_kv 2 → +**gqa_ratio 2 at head_dim 512**, a kernel-coverage hole on the 0.10.3 base: neither the MMA +(`switch_ncols2`, `fattn.cu`) nor the TILE (`switch_ncols2`, `fattn-tile.cuh`) D512 path instantiates +`ncols2 ∈ {1,2}` (gated `DV≤256` for binary size), so a D512/gqa≤2 `FLASH_ATTN_EXT` op aborts at runtime +(**bug-567**). Every other Gemma-4 config is gqa≥4 (A4B/12B/31B + E4B base; A4B globals are head16/kv2 → +gqa 8) and unaffected. + +`#491` was a host stop-gap: a `build_attn_mha` guard routed the D512/gqa≤2 layer to the existing non-FA +explicit-softmax branch (correct, but "fa off" on the hot path). `#493` is the real fix — a genuine +D512 f16 flash-attn via the VEC kernel — and **retires** the `#491` non-FA route for the f16 case. + +**Vendored-source markers `// opencoti #491` / `// opencoti #493`** (5 files, all nested; **not** an +`opencoti-hook:` — additive vendored-source edits, registered here): + +- `ggml/src/ggml-cuda/template-instances/fattn-vec-instance-f16-f16.cu` + `…/generate_cu_files.py` — + `DECL_FATTN_VEC_CASE(512, GGML_TYPE_F16, GGML_TYPE_F16)` instance + the autogen-emitter line. +- `ggml/src/ggml-cuda/fattn-vec.cuh` — `extern DECL_FATTN_VEC_CASE(512, F16, F16)`. +- `ggml/src/ggml-cuda/fattn.cu` — **two** head-size switches both get the D512 f16 case: the selector + (`ggml_cuda_get_best_fattn_kernel`: `Q->ne[0]==512 && gqa_ratio≤2 && gqa_opt_applies && K/V f16 → + BEST_FATTN_KERNEL_VEC`) **and** the VEC dispatch (`ggml_cuda_flash_attn_ext_vec`, `FATTN_VEC_CASE(512, + F16, F16)` before the `GGML_ABORT`). Patching only the selector makes it pick VEC while the dispatch + falls through to the abort → a **decode-time** `fattn.cu` fatal (boot is clean; it crashes on the first + global-layer decode). Both are required. +- `src/llama-graph.cpp` — narrow the `#491` guard: `fa_no_gpu_kernel` now requires K/V **not** f16/f32 + (`fa_d512_f16_vec`), so f16 D512/gqa≤2 flash-attns on-GPU and only quantized/bf16 D512/gqa≤2 (which the + assistant drafter never produces) keeps the soft_max fallback. + +The VEC kernel tiles `Q->ne[1]` (`cols_per_block` 1 decode / 2 prefill+verify), so the single f16 D512 +instance is a real flash-attn at **any n_q** (decode + prefill + the spec-verify batch). `logit_softcap` +is forced off for `D∉{128,256}` in the VEC kernel — safe because Gemma-4 `attn_soft_cap` is false. + +**Gate** (`.opencoti/m493-fa-d512-gate.sh`): E4B `-fa on` assistant-MTP decodes with **no abort** (147.86 +tok/s); the verified-greedy output is **byte-identical** to the `#491` soft_max path and draft acceptance +is unchanged (**0.67910** both) — assistant-MTP is exact spec-decode, so a wrong FA cannot corrupt the +verified output but *would* collapse acceptance; equal acceptance + byte-equal prefix ⇒ the VEC FA is +numerically correct. A4B (gqa 8, untouched MMA) regresses nowhere. Both DSO paths rebuilt byte-identical +(sha256 `a8ddcd82`). **Chain proof:** snapshot-diff capture of 5 files (0 new) from the post-`0079` base +(vendor backup `20260618-054922`, a real `#491=0/#493=0` post-0079 tree); `git apply --check` clean + +forward-reproduces live byte-for-byte. Touches `.cu`/`.cuh` ⇒ CUDA DSO rebuild + dual restamp required to +land live. **Applies after `0079`.** Provenance: opencoti-original. + +### `0081-tcq-q6-asym-kv` — TCQ KV tiers + q6_0 scalar KV + asymmetric TCQ FA-VEC (#447/#511/#512/#513/#515) + +Three additive KV-quant features landed as one snapshot-diff patch (they share the FA-VEC instance surface +and a single host enum bump, so they cannot be split without an intermediate `GGML_TYPE_COUNT` skew): + +1. **TCQ tiers** — `GGML_TYPE_TURBO3_TCQ = 48` (3.25 bpv; 512-state Viterbi, k=3/L=9) and + `GGML_TYPE_TURBO2_TCQ = 49` (2.25 bpv; 256-state, k=2/L=8). Trellis-Coded Quantization: a GPU FWHT-whitened + **Viterbi-trellis encode** (`turbo-tcq-cuda.cuh` `k_set_rows_turbo{3,2}_tcq`, dispatched from `set-rows.cu`) + + an **O(1) in-register codebook decode** fused into FA-VEC (`fattn-tcq.cuh` readers, smem codebook headers + `tcq-decode-cb.cuh`). Codebooks are real-27B-trained (rebuild-free reloadable). 27B KLD **0.0366** at + 3.25 bpv beats plain turbo3_0 **0.0470** at iso-bits — TCQ earns its keep on the head_dim-128 target. +2. **q6_0** — `GGML_TYPE_Q6_0 = 50` (6.5 bpv), a scalar KV type ported from `Anbeeld/beellama.cpp` (⊃ + `/shared/dev/ik_llama.cpp`, `block_q6_0 {ggml_half d; uint8_t qh[8]; uint8_t qs[16];}` QK6_0=32). 27B KLD + **0.01973** == q5_0 — it sits on the K-precision-dominant frontier (asymmetric strong-K combos like + `q8_0/q6_0` beat symmetric at equal size). `GGML_TYPE_COUNT` bumps 48 → **51**. +3. **Asymmetric TCQ** — the mixed `turbo3_tcq↔turbo2_tcq` FA-VEC instances (both directions), strong-K/cheap-V. + +**9 keeper FA-VEC instances** (the Anbeeld set, not the full cross-product, to bound compile blowup): +q6_0 ×5 (`q6_0/q6_0`, `q8_0/q6_0`, `q6_0/q5_0`, `q6_0/f16`, `f16/q6_0`) + TCQ ×4 (`turbo3_tcq`/`turbo2_tcq` +symmetric + both asymmetric). All read through **GPU FA-VEC** — `GGML_CUDA_FA_ALL_QUANTS` is the shipped +default (#531, `build-pipeline.ts --fa-all-quants`), and for the non-all-quants build the 9 instances are +force-compiled by `llamafile/build-functions.sh collect_gpu_sources` blocks 3b/3c/3d (the **#468/bug-339** +dlopen-symbol guard: without them the DSO lacks `ggml_cuda_flash_attn_ext_vec_case` and dlopen +fails the instant such KV reaches flash-attention; each is `[ -f ]`-guarded → no-op on a vanilla tree). A +`TURBO_TCQ_DUMP_ACTS=` env hook (#515) dumps post-FWHT whitened activations for rebuild-free codebook +retraining (zero overhead unset). + +The earlier `turbo3_tcq_w` 1024-state free-widen tier (#514) was measured **inert** and **removed** (#527) — +it is **not** in this patch (no `_w` enum/instances/kernels). + +**Vendored-source markers `// opencoti #447` / `#511` / `#512` / `#513` / `#515`** (38 files, all under +`vendors/sources/llamafile/` — **not** `opencoti-hook:` markers; these are additive vendored-source edits, +registered here, and do **not** change the `git grep "opencoti-hook:" packages/ docs/` count): + +- **9 NEW** `ggml/src/ggml-cuda/template-instances/fattn-vec-instance-{q6_0-q6_0,q8_0-q6_0,q6_0-q5_0,q6_0-f16, + f16-q6_0,turbo3_tcq-turbo3_tcq,turbo2_tcq-turbo2_tcq,turbo3_tcq-turbo2_tcq,turbo2_tcq-turbo3_tcq}.cu`. +- **3 NEW** `ggml/src/ggml-cuda/{fattn-tcq.cuh, tcq-decode-cb.cuh, turbo-tcq-cuda.cuh}`. +- Host enum/traits/quants: `ggml/include/ggml.h`, `ggml/src/{ggml.c, ggml-common.h, ggml-quants.c, + ggml-quants.h, ggml-turbo-quant.c}`, `ggml/src/ggml-cpu/{ggml-cpu.c, quants.h}`, `common/arg.cpp` + (`kv_cache_types += {Q6_0, TURBO3_TCQ, TURBO2_TCQ}`), `tools/perplexity/main.cpp`. +- CUDA glue: `ggml/src/ggml-cuda/{common.cuh, convert.cu, cpy.cu, cpy-utils.cuh, dequantize.cuh, fattn.cu + (FATTN_VEC_CASES + supports_op), fattn-common.cuh, fattn-vec.cuh, ggml-cuda.cu (SET_ROWS supports_op), + set-rows.cu}`, `ggml/src/iqk/{iqk_common_extra.h, iqk_ggml_type_ext.h}`. +- Graph/DCA lift: `src/{llama-graph.cpp, dca.cpp}` (scalar q6_0 + asym compose with DCA via the #444 + f16-dequant-on-lift; **TCQ aborts under DCA**, rc=134, pending #505). +- Build glue: `BUILD.mk`, **outer** `llamafile/build-functions.sh`. + +**Correctness** via KLD / logit-equiv / RULER niah — never greedy needle (bug-263/270). RULER niah 8 tiers +**100/100** @ 4k–512k (#529 §5b). PPL/KLD valid **only** on Qwen3.6-27B-Omnimerge (head_dim 128); A4B raw-PPL +is garbage (bug-631). TCQ pays an in-register-Viterbi decode tax (~15% slower tg on weight-bound 27B, ~45% on +small A4B). **Chain proof:** snapshot-diff capture (38 files, 9 NEW) from the post-`0080` BASE (vendor backup +`20260622-111006-pre-tcq-q6-asym-capture`); applying `0081` to a clean `apply(0001..0080)` tree +**byte-identically reproduces GOAL** (full-tree `diff -rq` empty, nested + outer). Touches `.cu`/`.cuh` ⇒ CUDA +DSO rebuild + dual restamp required to land live. **Applies after `0080`.** Provenance: opencoti-original +(q6_0 ported from Anbeeld/beellama.cpp ⊃ ik_llama.cpp). + +### `0089`/`0093`–`0095` — bug-858 MTP parity ship + bug-2108 chain repair (2026-07-05, #608/#614) + +**bug-2108 chain repair (renumbering).** The committed chain no longer +strict-applied from clean 0.10.3: `0089-rolling-kv-compute-reserve` (and later +patches) were snapshot-captured from a tree carrying **uncaptured** work +(sparse-attn vslash #551/#557–568 incl. the `sparse_attn_block_size` KV-ctor +plumbing, #591/#592 decode-overhead work, bug-1840 fixes), so their context +lines baked in content no patch introduced. Repair: +`0089-sparse-attn-vslash-foundation` (NEW — `diff(chain-0088 baseline, +vendor-backup 20260701-123044)`, proven to be exactly old-0089's capture base); +old `0089`→`0090`, `0090`→`0091`; old `0091-rolling-kv-diag-visibility` +**retired** (its `llama-kv-cache.cpp` hunks folded into `0093`). Full-chain +proof: clean `git archive` 0.10.3 + Mozilla `llama.cpp.patches` overlay + +strict `git apply` of all 41 patches ⇒ **byte-identical** to the live vendored +tree (excl. `tools/ui/dist` fetch-time assets + submodules). Lesson (bug-250 +class): run the chain proof at EVERY capture, not once per era. + +**Markers registered by this ship:** + +- `// opencoti-hook: sparse-attn-vslash` (`common.h` + graph/KV plumbing) — + vertical-slash sparse attention foundation; patch `0089`. +- bug-858 dual-ctx comment blocks (`// opencoti bug-858 dual-context MTP …`) + across `speculative.cpp`, `llama-context.*`, `llama-kv-cache*`, + `llama-model.cpp`, `models/gemma4-assistant.cpp`, `server-context.cpp` — the + `ctx_other`/`mem_other`+`share` drafter-shares-target-KV port; patch `0093`. + Control-plane stays opencoti (`--spec-type draft-assistant --mtp-head` + + `OPENCOTI_MTP_DUAL_CTX=1`); data plane aligned with upstream #23398/#24282. +- `// opencoti-hook: mtp-verify-mmvf (bug-2103)` (`ggml-cuda/mmvf.cu`) — + TinyBLAS-gated `ne11≤8` mmvf cap for the spec-verify GEMV cliff; patch `0093`. +- `// opencoti-hook: tinyblas-small-gemm (bug-2104)` (`llamafile/tinyblas.cu`) + — second small-GEMM config when the 128×64 config yields <24 blocks; patch + `0094`. NOTE: outer-llamafile file (not llama.cpp) — survives llama.cpp + bumps, conflicts only on llamafile-proper bumps. +- `// opencoti-hook: ckpt-buf-noinit (#614/bug-2107)` (`common.h`, + `server-task.h`, `server-context.cpp`) — `common_state_buf` + checkpoint + buffer recycle pool; rides with `checkpoint_min_step` 256→8192 (bug-2105, + plain default alignment, no marker — registered via this row); patch `0095`. + Future-bump gotcha (bug-2106): the struct default is header-inlined into + EVERY constructing TU (incl. `o/llamafile/*.o`) — after editing it, purge all + constructing objects (both arches, `CCACHE_DISABLE=1`) and gate on the boot + line `min spacing = 8192`, never on "build succeeded". + +### `0096` — single-context Gemma MTP facade REMOVED (#611, 2026-07-05) + +END of the bug-858 port. The dual-context assistant (patch `0093`) is now the +ONLY `--spec-type draft-assistant` execution path — the `OPENCOTI_MTP_DUAL_CTX` +env gate and the whole single-context in-target engine are deleted (~1600 +lines: `decode_mtp*` facade/sync/fused + async worker, `process_ubatch_mtp`, +`sched_mtp`, `llama_decode_mtp` C API, iswa `init_mtp`, `build_attn_mtp`, +`llm_build_gemma4_mtp`, `common_speculative_impl_draft_assistant`). + +**Registry deltas:** the `opencoti F5 M6-S4 mtp` markers covering those +functions are gone with them; the surviving assistant surface is the loader +(`llama_model_load_mtp_from_file` / `mtp_assistant`), the dual-ctx graph in +`models/gemma4-assistant.cpp`, and the `0093` dual-ctx comment blocks. The +NextN fused-draft markers (`fused-nextn-mtp`, `mtp-verify-mmvf`) are untouched. +A future llamafile bump reconciles the assistant almost entirely against +upstream #23398/#24282 (data plane AND execution shape now upstream-aligned; +remaining opencoti-specific: control-plane flags + single-binary loading). + +Gate: A4B n2 305.4 tps / 0.849 accept == 0093 reference (boot line +`dual-context draft created` without any env); q35 NextN n3 248.8 / 0.662. +bug-2110 (apparent accept 0.635 at `--parallel 2 --kv-unified`) was RESOLVED +same-day as a tps-harness artifact (degenerate-filler prompt): on the real +9-prompt mtp-bench the dual-ctx assistant holds accept at par2 (ours +0.785/0.783 par1/par2; upstream 0.793/0.792) — PolyKV × assistant-MTP composes +cleanly on the dual-ctx path. + +## 2026-07-05 — 0097 native-ELF CMake lane (#619, secondary artifact of #613) + +`0097-native-elf-cmake.patch` makes the vendored tree buildable as a standard +Linux ELF `llama-server` (plain cmake + g++/nvcc, rpath `libggml-cuda.so`, +clean gdb/nsys) alongside the untouched cosmocc Makefile lane. + +Registry additions (all markers `opencoti-hook: native-elf-cmake (#619)`): + +- `llama.cpp/ggml/src/CMakeLists.txt` — add `ggml-turbo-quant.c`, + `ggml-neo-pipeline.cpp`, `ggml-iqk-flash-attn.cpp` to ggml-base sources. +- `llama.cpp/src/CMakeLists.txt` — add `dca.cpp` (matches neither the + `llama-*.cpp` list nor the `models/*.cpp` glob). +- `llama.cpp/common/CMakeLists.txt` — add `pcie-profile.cpp`. +- `llama.cpp/ggml/src/ggml-cpu/CMakeLists.txt` — add + `ggml-cpu/opencoti-elf-stubs.cpp` (CMake-lane-only). +- `llama.cpp/tools/server/CMakeLists.txt` — + `set_source_files_properties(server.cpp … main=llama_server)`: the Mozilla + overlay's server.cpp defines `int main` where upstream defines + `llama_server`; the cosmo lane renames at compile time, CMake must too. +- NEW FILE `llama.cpp/ggml/src/ggml-cpu/opencoti-elf-stubs.cpp` — return-false + stubs for the `llamafile/sgemm.cpp` dispatchers (`llamafile_mixmul`, + `_mixmul_needs`, `_mixmul_iqk`, `llamafile_fa_*`); every call site treats + false as "use the upstream generic path" (same contract as Mozilla's + `fa_helpers_unsupported.cpp`), so the ELF build loses only CPU fastpaths. + sgemm.cpp itself includes `` and stays cosmo-lane-only. + +Bump guidance: on a future llamafile bump, re-anchor the five one-line list +insertions (trivial context) and keep the stub file as-is unless sgemm.cpp's +dispatcher signatures change. The CMake lane REQUIRES +`-DGGML_CUDA_FA_ALL_QUANTS=ON` — upstream's CMake only globs +`fattn-vec*.cu` (our turbo/TCQ instances + codebook setters) under that flag; +the cosmo build always sets it. CMake lane uses cuBLAS (not TinyBLAS) — +per-toolchain perf gating (bug-2103/2104 class). + +Chain state after 0097: 43 patches, strict-apply + byte-identical replay +proven from clean 0.10.3 archive + Mozilla overlay on 2026-07-05. + +## 2026-07-05 — 0098 rolling-KV lse-decode (bug-1843, #623) + +`0098-rolling-kv-lse-decode.patch` extends the Stage-3a LSE channel +(`opencoti_fattn_dst_lse` / `opencoti_fattn_dst_lse_written`, both defined in +`fattn-common.cuh` and drained by `launch_fattn`) to `head_dim<=256` `n_q==1` +decode at the three streaming-FA call sites in +`llama.cpp/ggml/src/ggml-cuda/fattn.cu` (markers: `opencoti-hook: +f5-rolling-kv bug-1843` at the S3b overlap site; short `bug-1843:` notes at +the S3a single-stream and S2 resident-view sites). The per-tile +`streaming_lse_kernel` recompute now fires only when +`opencoti_fattn_dst_lse_written` reports that no finalize (stream-k fixup / +parallel-blocks combine) ran. No header change; no new symbols; DSO-only +rebuild. Chain state after 0098: 44 patches, strict-apply + byte-identical +replay proven from clean 0.10.3 + Mozilla overlay (2026-07-05). + +## 2026-07-06 — 0099-0102 DCA-Gemma wiring + attn-rot inherit + streaming scratch ring (bug-2118/2119/2120/2121) + +Four fixes captured after the Gemma×DCA contamination discovery (bug-2118) and +the two criticals it surfaced: + +- `0099-dca-gemma4-wiring.patch` — surgical hooks in `llama.cpp/src/models/gemma4.cpp` + + `gemma4-assistant.cpp`: `build_attn_inp_dca` + per-layer `dca_l` dispatch to + `build_attn_dca` (GLOBAL layers; assistant Q-only). Marker: `opencoti F5 dca`. +- `0100-dca-single-chunk-defer.patch` — `llama.cpp/src/dca.cpp` (opencoti-owned file): + single-chunk regime defers quant lift like analytical-bands. +- `0101-kv-attn-rot-inherit.patch` — `llama.cpp/src/llama-kv-cache.cpp` ctor share + branch inherits attn_rot_k/v + Hadamard tables from the source cache. Marker + comment: `opencoti bug-2120`. +- `0102-fattn-bulk-h2d-scratch-ring.patch` — `llama.cpp/ggml/src/ggml-cuda/fattn.cu`: + bug-2116 bulk-H2D quant tail lift + bug-2121 persistent per-slot staging scratch + ring (markers: `opencoti-hook: f5-rolling-kv bug-2116` / `bug-2121`). RULE banked in + cerebrum: never share ggml_cuda_pool scratch across CUDA streams. + +Chain state after 0102: 48 patches; full chain strict-applies from clean 0.10.3 +pin (7fca8b2 + nested dbe9c0c8 + Mozilla overlay) and reproduces the live tree +byte-identically on all touched files (verified 2026-07-06, capture-b21xx). + +## 2026-07-07 — 0103-0104 rolling-KV CPU-FA probe guard + q6_0/q4_0 FA-VEC cell (bug-2125 / #627) + +Two fixes for the DCA + custom-KV boot SIGSEGV / abort found on bs2 at 256k/512k: + +- `0103-kv-cpu-fa-probe-guard.patch` — `llama.cpp/src/llama-kv-cache.cpp`: gates + the Stage-3d-S2 tactic selector's synthetic **CPU** flash-attn companion probe + (`opencoti_fa_compute_probe_ms` with `on_cpu=true`) behind a new host predicate + `opencoti_cpu_fa_capable_ktype`. That probe modeled an unused CPU_FA_TAIL cost; + for CUDA-FA-VEC-only KV types (q6_0, turbo{2,3,4,8}_0, turbo{2,3}_tcq) the CPU + vec_dot trait is NULL, so building/running the host flash-attn NULL-derefed → + SIGSEGV at context init (before any request). The predicate returns 0.0f for + custom-K tiers, leaving CPU_FA_TAIL unmodeled — the selector keeps its pure-GPU + POSITION_WINDOW default (**no** CPU fallback: the rolling window stays fully on + GPU for every KV tier). Marker comment: `opencoti bug-2125`. Host-only. +- `0104-fattn-q6q4-vec-instance.patch` — `llama.cpp/ggml/src/ggml-cuda/fattn.cu` + + new `template-instances/fattn-vec-instance-q6_0-q4_0.cu` + `llamafile/build-functions.sh`: + adds the q6_0-K / q4_0-V FA-VEC cell (strong-K, cheap-V — the K precision ≥ V + invariant) missing from the dispatch matrix, which crashed the D≤256 path + (Gemma-4 local/SWA) into `GGML_ABORT`. Marker: `opencoti-hook: q6_0 KV (#627/bug-2125)`. + The reverse q4_0/q6_0 direction is intentionally NOT added (K must stay + higher-bit). CUDA DSO change. Mirrors the q6_0-q5_0 pattern (#512). + +Chain state after 0104: 50 patches; full chain 0001..0104 strict-applies from +clean 0.10.3 pin and reproduces the live tree byte-identically on all four +touched files (verified 2026-07-07, capture-b2125). Gates (bs2 256k DCA-on): +q6_0 / turbo3_tcq / turbo2_tcq symmetric + q6_0-K/q4_0-V all boot HEALTHY + +niah_single_1 = 100.0 (every one previously SIGSEGV/abort at boot). + +## 2026-07-09 — 0105-0107 WS1 asym scalar-K/turbo-V ship (bug-2126/2127/2130, #628/#630) + +**No new opencode source markers.** WorkStream-1 (asymmetric strong scalar-K + +cheap turbo/TCQ-V decode) lands entirely in the **vendored** `llama.cpp` tree as +three additive patches; the packages/ Registry above is unchanged. Split: +`0105` DCA per-tensor defer-decouple (bug-2127, dca.cpp, host-only, the ~2× +decode win); `0106` scalar-K/turbo-V boot-correctness guards (bug-2126, +llama-kv-cache.cpp, host-only, two null-fn-ptr SIGSEGV fixes in the rolling-KV +boot probe); `0107` turbo-MMA verify kernels (D128/D256/D512, OPT-IN default-OFF) ++ the 16 fused-VEC scalar-K/turbo-V decode instances + fattn.cu selector + +llama-graph.cpp fuse path + bug-2130 `need_f16` (fattn-mma-f16.cuh) + +build-functions.sh default-build inclusion. The dropped WS1 option-2 boot +"fuse-autoselect" scaffolding (`opencoti_turbo_fuse_autoselect`, +`g_opencoti_asym_fuse_enable[2][2]`, `probe_nq`/`materialize` probe params) is +**not** in any patch — it was stripped at ship (inert, default-fuse-equivalent). +The WS1 `// opencoti-hook: WS1 …` markers live in vendored source only (not +counted by the packages/ conformance grep). Per-file registry: the +`vendors/patches/llamafile/README.md` `0105`/`0106`/`0107` rows. + +turbo-MMA VERDICT (#630, 3-model crossover A4B/omni27/qw35a3): earns no +default-on anywhere → shipped OPT-IN default-OFF (Gemma A4B/D256 wins ≤128k, +net-loses @256k + un-gatable windowed; Qwen/D128 bit-exact but tps-flat, +FFN-bound). Chain state after 0107: 53 patches; full chain 0001..0107 +strict-applies from clean 0.10.3 pin and reproduces the intended FINAL tree +byte-identically (6 modified + 19 new TUs match; llama-kv-cache.h returns to +baseline; zero scaffolding symbols) — verified 2026-07-09, capture-ws1-ship. + +## 2026-07-10 — 0108-0109 WS2 turbo-V in-register DCA-decode ship (bug-2134/2140, #629) + +**No new opencode source markers.** WorkStream-2 (read a rotated turboN V-cache +IN-REGISTER inside the multichunk DCA fused flash-attn kernel, killing the +per-forward whole-cache `cpy_turboN_f16` cast) lands entirely in the **vendored** +tree as two additive patches; the packages/ Registry above is unchanged. Split: + +- `0108-ws2-dca-fused-turbo-v-inregister.patch` (bug-2134) — `llama.cpp/src/dca.cpp` + (the WS2 in-register gate, now **DEFAULT-ON**; `WS2_DCA_TURBO_V=0` is the + f16-lift escape; n_q≤`WS2_NQ_MAX`(16) gate keeps large-n_q PREFILL on the + byte-identical f16-lift path) + `ggml/src/ggml-cuda/fattn-mma-f16.cuh` + (D512+D256 turbo/tcq `dca_fused` instance macros + the TCQ `to_fp16_nc` assert + exemption for the in-register-V case) + `ggml/src/ggml-cuda/fattn.cu` (D256 + `case 256` turbo2_0/turbo3_0/turbo2_tcq/turbo3_tcq V-type dispatch) + + `template-instances/generate_cu_files.py` + the new turbo/turbo-D256/turbo2tcq-D256/ + turbo3tcq-D256 `dca_fused` TU shards. Markers: `opencoti-hook: WS2 turbo-V DCA + (#629)` (host + kernel). bug-455 (dca_fused forces `Q_in_reg=false`) inherited + by the turbo D256 path. CUDA DSO + host change. +- `0109-cuda-dso-2gb-fatbin-last.patch` (bug-2140) — `llamafile/build-functions.sh` + + `llamafile/cuda.sh`: two build hooks. **`cuda-fatbin-last` (PRIMARY)** — the + ~1.9GB `.nv_fatbin` (6 SM arches × ~260 cubins) sat physically between host + `.text` and its `.bss`/`.got`, so every host device-stub PC32/PLT/TLSGD reloc + spanned >2^31 → link truncation once the D256 tcq TUs pushed the image past 2GB. + Root fix = force `.nv_fatbin` to the image END via an `INSERT AFTER .bss` + orphan-section linker fragment (pure `-Wl`, no code-model change; fixes ALL + reloc classes; the only fatbin ref left is a 64-bit data pointer). **`cuda-mcmodel-large` + (OPT-IN, default `none`)** — `OPENCOTI_MCMODEL=large|medium|none` forces the large + code model on host x86 (compile via nvcc `--compiler-options`, link via + `-Xcompiler`) + `-Xlinker=--no-relax` for the prebuilt-CRT/`libcudart_static` + GOTPCRELX; superseded by fatbin-last, kept as a documented escape. Device SASS + untouched → zero kernel-perf cost (perf-regression A/B confirms parity). Markers: + `opencoti-hook: cuda-fatbin-last` + `opencoti-hook: cuda-mcmodel-large`. Build-flag/ + linker change only. Per-file registry: the `vendors/patches/llamafile/README.md` + `0108`/`0109` rows. + +WS2 VERDICT (#629, 4-arch × 4-tier matrix @256k DCA-on): **default-ON everywhere** — +in-register turbo-V is a validated decode WIN of +68-95% (D512 A4B/31B, D256 +omni27/qw35a3 × turbo2_0/turbo3_0/turbo2_tcq/turbo3_tcq), lossless by RULER niah +(100) + teacher-forced logit-equiv (real_frac 0 / ffa 1.0), incl. A4B MTP-on +n_q>1 verify (real_frac 0, mean_tv 2e-05). Opposite of the WS1 turbo-MMA null: +WS2 removes an O(n_kv) whole-cache cast that grows with ctx, so it wins on the +serving decode path. Chain state after 0109: 55 patches; full chain 0001..0109 +strict-applies from clean 0.10.3 pin and reproduces the live WS2 tree +byte-identically — verified 2026-07-10, capture-ws2-ship. + +## 2026-07-10 — 0110 WS2 asym-band NaN guard (bug-2141, #629/#626) + +**No new opencode source markers.** The WS2 (0108) in-register turbo-V dca_fused +decode reads V rotated and de-rotates the graph OUTPUT via `wht_o`; the normalized +`O = VKQ / rowsum` is produced in the shared stream-k combine. Under DCA-on decode +the fused kernel walks INTER→SUCC→INTRA phases under one continuous online-softmax +and load-balances via the stock kbc stream-k decomposition, so the final +divide-by-rowsum lands in the shared stream-k fixup for seam blocks (and the +process_tile complete-tile write-back otherwise). A band-boundary phase with an +all-(−inf) mask gives a ZERO exp-sum (rowsum==0) → 0/0=NaN, which wht_o's 128-pt +butterfly smears across all 512 de-rotated dims → NaN logits → empty generation. +Surfaces ONLY on asymmetric turbo_tcq K≠V (the hi-K/cheap-V serving pattern): +t3tcqK/t2tcqV + t2tcqK/t3tcqV pre-fix niah 20 / 4-of-5 EMPTY @256k/8-chunk (already +100 @512k/16-chunk) — a composition no symmetric control exercises. Discriminator +`WS2_DCA_TURBO_V=0` (V→f16-lift, same turbo3 K-lift) recovers to niah 100 / 0-empty, +isolating the fault to the in-register-V + wht_o composition (NOT the K-lift/codebook +— all per-tensor type selection verified correct). + +- `0110-ws2-asym-band-nan-guard.patch` (bug-2141) — guards the zero exp-sum divide + at the three FA output-normalization sites feeding wht_o: + `ggml/src/ggml-cuda/fattn-mma-f16.cuh` (`flash_attn_ext_f16_process_tile` + complete-tile write-back) + `ggml/src/ggml-cuda/fattn-common.cuh` + (`flash_attn_stream_k_fixup_uniform` + `flash_attn_stream_k_fixup_general` + stream-k seam combines). Emits the identity 0 for a fully-masked row; the + `rowsum!=0` branch runs the EXACT original division (not a reciprocal-multiply) + → bit-identical for all symmetric turbo/TCQ + scalar tiers and all non-DCA + callers of these shared kernels; the WS2 decode win is untouched (no-op on the + hot path). Markers: `opencoti-hook: WS2 asym-band NaN guard (bug-2141)` (3 sites, + kernel-only). CUDA DSO change; no host binary change. Per-file registry: the + `vendors/patches/llamafile/README.md` `0110` row. + +RE-GATE VERDICT (#632, fixed DSO f1c554d8): full asym turbo_tcq matrix RECOVERED — +t3tcqK/t2tcqV AND t2tcqK/t3tcqV at 256k AND 512k all niah **100.0 / 0-empty** +(pre-fix t3tcqK/t2tcqV @256k was 20 / 4-of-5 empty). Symmetric controls +(turbo2_tcq, turbo3_tcq @256k) hold **100 / 0-empty**. WS2 in-register decode win +preserved through the guard (A4B turbo2_0 @256k warm **~91.2 tps**, uncontended). +Off-path bit-identicality confirmed by the symmetric/scalar controls staying 100 +plus the guard's rowsum!=0 branch running the identical division. Chain state after +0110: 56 patches; full chain 0001..0110 strict-applies from clean 0.10.3 pin and +reproduces the live tree byte-identically — verified 2026-07-10 +(fattn-mma-f16.cuh 58cdae34, fattn-common.cuh f7af3a8d). + +## 2026-07-10 — 0111-0113 rolling-KV spill made graceful (bug-2144/2145/2146, #635/636/637/631) + +Three host-only additive patches that make rolling-KV overflow (POSITION_WINDOW KV +residency) actually graceful — the "spill fallback" now degrades on the PCIe/bytes +axis instead of cliffing. All validated on the same binary (#637, Qwen2.5-14B-1M Q8_0 +@128k, bs2 RTX 6000, single-GPU, needle PRESENT every cell); the DSO is unchanged +(the bug-2144 kernel tail-read `src[7]/[8]` already shipped in the C2 DSO with 0108). +New markers live in vendored source only; the packages/ Registry is unchanged. + +- `0111-dca-streaming-tail-bug2144.patch` (bug-2144) — `llama.cpp/src/dca.cpp`. DCA-on + rolling-KV spill decode cliffed (~55×) because `build_attn_dca_core` used the + whole-cache `get_k`/`get_v`, which for a spilling POSITION_WINDOW layer return the + `ggml_concat(window, tail)` fallback (sync full-KV round-trip every step), bypassing + the async streaming path. Fix gives the fused `dca_fused` kernel a two-region + window+tail streaming path (window on GPU, pinned-host tail on `src[7]/[8]`, + `op_params[5]=n_win`), gated `dca_window_decode` (FA ∧ small-n_q ∧ !single_chunk ∧ + `!get_is_fragmented()` ∧ POSITION_WINDOW active ∧ window active). **DEFAULT-ON** + (escape `WS2_DCA_STREAM=0`); safe because the master flag only arms the path and the + predicate enforces every precondition structurally. Markers: + `opencoti-hook: DCA streaming-tail (bug-2144-C / Stage C)` (2 sites). Host-only. +- `0112-rolling-kv-phasegate-prefill-bug2145.patch` (bug-2145) — + `llama.cpp/src/llama-graph.cpp`. The non-DCA streaming-window path has no batched + large-n_q kernel → PREFILL ran decode-style over the window = flat ~325 tok/s cliff. + Fix gates the window fast-path to decode/small-n_q (`WS2_NQ_MAX` d16) and routes + large-n_q prefill to the amortized whole-cache concat + batched `build_attn_mha` + (path B), at both dispatch sites. Markers: + `opencoti-hook: rolling-KV phase-gate prefill (bug-2145)` (3 sites). Host-only. +- `0113-rolling-kv-reserve-fixpoint-bug2146.patch` (bug-2146) — + `llama.cpp/src/llama-context.cpp`. The bug-1342 two-pass reserve measured the compute + buffer against a degenerate `wc=256` window; with the 0111 streaming decode path that + forces a maximal tail whose scratch inflated the measurement (~4088 vs real ~926 MiB + → 3.16 GiB over-reserve, a fitting KV pushed to a 3168 MiB tail). Fix iterates the + reserve to a fixpoint (cap 3 iters) — each re-size grows the window/shrinks the + tail/shrinks the buffer, converging to full residency when the KV fits; non-streaming + converges in one step byte-identical to prior. No new marker (the fixpoint loop is + self-documented `bug-2146`). Host-only. + +VALIDATED (#637): DCA-on streaming spill tracks-and-beats the DCA-off rolling-KV +reference at every matched tail — plateau ~37 tps to a ~4.9 GiB tail, common knee +~5.6 GiB, graceful rolloff, NO cliff; ~17× recovery at a 6.7 GiB tail (0.68→11.69). +Reserve fixpoint recovers full residency (37.10→38.40 @vt=41888) and improves the +spill case (+62% @vt=35400). Default-on re-gate (no env) reproduces the numbers. +Chain state after 0113: 59 patches; full chain 0001..0113 strict-applies from clean +0.10.3 pin and reproduces the live tree byte-identically — verified 2026-07-10 +(dca.cpp efa73317, llama-graph.cpp 874fa901, llama-context.cpp e40aa9e2). Per-file +registry: the `vendors/patches/llamafile/README.md` `0111`/`0112`/`0113` rows. + +## 2026-07-10 — 0114 hybrid rolling-KV residency + 0111 ggml host-op amendment (bug-2142, #633) + +- `0114-hybrid-rolling-kv-residency-bug2142.patch` (bug-2142) — + `llama.cpp/src/llama-memory-hybrid.{h,cpp}`, `llama-memory-hybrid-iswa.{h,cpp}`, + `llama-model.cpp`. The rolling-KV residency knobs (`--vram-target`, + `--kv-residency-mode`, the bug-1342 compute-reserve, the window-measure-pass) reached + the iSWA and plain-dense memory paths (llama-model.cpp `create_memory`) but were + **silently dropped** for hybrid models: `llama_memory_hybrid` and + `llama_memory_hybrid_iswa` hardcoded `0/0.0f/false` when building their internal + attention `llama_kv_cache`, so a hybrid model (qw35a3 = Qwen3.6-35B-A3B DeltaNet+attn + NextN MoE) on a too-small card ignored `--vram-target` and OOMed. Fix threads the five + residency params through both hybrid wrapper ctors to the attention cache, mirroring + the iSWA/plain forwarding; the new ctor params default to the prior off-state so any + non-forwarding caller and every resident config stays byte-identical. Markers: + `opencoti-hook: rolling-KV residency for hybrid (bug-2142)` (2 `create_memory` call + sites in `llama-model.cpp`, one per hybrid variant). Host-only — no DSO change. + VALIDATED (#633, bs2 GPU0): fixed `--vram-target 32000` @ `-c 262144` on qw35a3 → + `host tail = 4705 MiB` (knob honored); pre-fix (C2ship) same cmd → `FULLY RESIDENT … + no host tail` (dropped, GPU overshoots the 32000 target to 34907 MiB). Fully-resident + decode unchanged (182.5 t/s); spill decode 77.8 t/s (bandwidth-bound, as expected). + +- **0111 amendment (bug-2144 completeness).** The originally-shipped + `0111-dca-streaming-tail-bug2144.patch` captured only the `dca.cpp` call-site + (`kp_tail`/`vp_tail`/`dca_n_win`) but NOT the matching host-side op-node construction + that call needs to compile — the `k_tail`/`v_tail`/`n_win` parameters on + `ggml_flash_attn_ext_dca_fused` and their `src[7]`/`src[8]`/`op_params[5]` wiring in + `ggml.h`/`ggml.c`. The shipped C2 binary built from a working tree that had those + edits, so a strict re-apply of the chain would REVERT them and fail to compile; the + gap was masked because the chain proof did not build. 0111 is regenerated to include + the `ggml.h`/`ggml.c` hunks (recovered byte-identically from the pre-ship vendor + backup). Lesson banked: the chain-apply proof MUST include a build step so a + non-building chain can't hide an uncaptured edit. See buglog bug-2142/bug-2147. + +Chain state after 0114: 60 patches. Full-chain 0001..0114 strict-apply + build proof +is the gating step for this section (in progress at commit time). + +## 2026-07-11 — 0115 refuse context-shift on a spilled rolling-KV window (bug-2148, #638) + +- `0115-rolling-kv-shift-guard-bug2148.patch` (bug-2148) — + `llama.cpp/src/llama-kv-cache.cpp`, one hunk in `llama_kv_cache::get_can_shift()`. + The F5 M7 Stage-3c position window splits the cache across a GPU-resident window + `[0,wc)` and a CPU-resident tail `[wc,kv_size)` when `--vram-target` forces overflow. + Context-shift (server `seq_rm`+`seq_add`→`pos_add`) and self-extend (`seq_div`→`pos_div`) + re-rope the cache IN PLACE via the `llm_graph_input_k_shift` / `build_rope_shift` graph, + which views only `k_per_stream` — sized to the window's `wc` cells, NOT `get_size()` + (llama-kv-cache.cpp ~4720-4744). So the moment a shift fires with a spilled window it + over-reads the window tensor and hits `ggml.c` `GGML_ASSERT(view_src == NULL || data_size + == 0 || data_size + view_offs <= ggml_nbytes(view_src))`; and even were the view to fit, + the host tail (written already-roped, never transformed again — no CPU-side rope path) + would be left stale-roped. Fix: `get_can_shift()` returns `false` when any layer has an + active spilled window (`window_cells > 0` && a populated `k_cpu_per_stream`); the server + reads this at init (`common_init_from_params` → "KV cache shifting is not supported for + this context, disabling KV cache shifting"; also server-context.cpp:1093) and cleanly + disables `ctx_shift` + cache-reuse, so a spilled slot is bounded at `n_ctx` instead of + crashing. Fully-resident windows (`wc==kv_size`, no tail, `window_cells` sentinel 0) and + every non-window cache find no spilled layer → BYTE-IDENTICAL and still context-shift + normally; iSWA (`get_can_shift` ANDs kv_base+kv_swa) and hybrid (`get_can_shift` delegates + to `mem_attn`) wrappers propagate the leaf guard. The "no-degradation" alternative (a + host-side rope-by-delta pass over the CPU tail on every shift) is DEFERRED — out of scope + for the multi-session prefix-shared serving target, which does not rely on infinite-context + shift over a spilled tail. Marker: `opencoti-hook: rolling-kv-shift-guard` (1 site). + Host-only — no DSO change. VALIDATED (#638, bs2 GPU0, Qwen2.5-14B-1M-Q8_0, `-c 12288 + --context-shift --keep 256`): windowON (`--vram-target 18000`, 6912 resident / 5376 host + tail) → shift disabled at init, request bounded (predicted_n=3840, 8448+3840=12288), + coherent, NO assert; controlOFF (`--vram-target 80000`, fully resident) → shift STILL fires + (n_discard=6015), coherent to 1219 — guard does not over-fire. Prior binary asserted at + `ggml.c:1840` on the windowON shift. Byte-identical `git apply` of 0115 == the live edit. + +Chain state after 0115: 61 patches. + +## 2026-07-11 — 0116 auto KV-tier selection at boot (HAL #582 P1, #621) + +- `0116-kv-auto-tier-hal582-p1.patch` (#621) — the HAL #582 P1 policy: on a dense + full-attention model, auto-pick the LEAST-compressing scalar KV type pair that keeps the + whole cache resident, rather than making the operator hand-tune `-ctk`/`-ctv`. New static + helper `opencoti_auto_select_kv_tier` in `llama-kv-cache.cpp`, called in the + `llama_kv_cache` ctor BEFORE any residency/allocation math consumes `type_k`/`type_v`. It + reuses the EXACT budget arithmetic of `opencoti_compute_resident_window_cells` (free − + already-resident − compute-reserve, capped by `--vram-target`) and the same per-layer + per-cell byte sum, then walks the ladder `{f16, q8_0/q8_0, q8_0/q4_0, q5_1/q4_0}` (HAL §2 + hi-K/cheap-V recipe; `q5_1/q4_0` last-resort floor) and selects the first (least- + compressing) rung whose whole-cache cost fits; floor if none fit (rolling-KV then spills the + excess). Announced at WARN with a `set -ctk/-ctv to override` hint (mirrors the sparse-V + auto-policy #567). **Strictly opt-in** (`OPENCOTI_KV_AUTO_TIER=1`) and a no-op — byte- + identical — when the env is unset (every existing caller), there is no GPU offload, the user + set `-ctk`/`-ctv` off the f16 default (override respected), the model is iSWA + (`swa_type != NONE`, Gemma stays f16/resident per HAL §3c), or f16 already fits. +- Marker: `opencoti-hook: kv-auto-tier` (1 site — the ctor helper + call in + `llama-kv-cache.cpp`). Provenance: opencoti-original. +- Host adapter opt-in: `kvAutoTier` config → `OPENCOTI_KV_AUTO_TIER` env in `buildServerArgs` + (`packages/opencoti-llamafile`). Additive, no surgical hook. +- Host-only — no DSO change (make-only build, ~11 s ccache-warm). VALIDATED (#621, bs2 GPU0, + Qwen2.5-14B-1M-Q8_0, `-c 32768`, RTX 6000): ladder walks correctly as `--vram-target` + tightens — 20000 MiB → q8_0/q8_0 (KV 3264 MiB fits budget 3328), 19400 → q8_0/q4_0 (2496 + fits 2728), 18800 → q5_1/q4_0 (2016 fits 2128); every rung `FULLY RESIDENT … no host tail`, + passkey needle clean. Config-equivalence: auto q8_0/q8_0 output byte-identical to explicit + `-ctk q8_0 -ctv q8_0`, which emitted zero auto-tier WARN (opt-in path confirmed off). + Byte-identical `git apply` of 0116 == the live edit. + +Chain state after 0116: 62 patches. + +## 2026-07-11 — 0117 re-rope a spilled rolling-KV window on context-shift (#639, bug-2148 reopened) + +- `0117-rolling-kv-shift-rerope-639.patch` (#639) — supersedes 0115. 0115 stopped the + spilled-window shift from crashing by REFUSING it (`get_can_shift()` returned false when a + layer had an active spilled window), which cleanly but silently disabled `ctx_shift` AND KV + prompt-cache-reuse for any multi-session slot whose KV spilled — the exact serving-path + capability this fork exists to preserve. 0117 does the deferred hard part instead: + `build_graph_shift` re-ropes BOTH regions of a spilled window with the same `build_rope_shift` + op — the WINDOW on its GPU tensor `k_per_stream[s]` (a view bounded to `wc` cells, so it never + over-reads `get_size()`; shift slice `[base, base+wc)`) and the TAIL on its host tensor + `k_cpu_per_stream[s]` (a view of the `kv_size-wc` tail cells; shift slice `[base+wc, + base+kv_size)`). The tail tensor lives in host memory, so ggml's scheduler dispatches its rope + op to the CPU backend automatically — identical rope formula, one-shot on shift events only, + never per decode token — while attention still runs entirely on GPU over the already-roped + window+tail (no CPU-attention / CPU_FA_TAIL divergence class). `get_can_shift()` returns true + again (keeping the STEP35 and `n_pos_per_embd()>1` guards); the 0115 refusal loop is removed. +- Regression shield: fully-resident windows (`window_cells` sentinel 0) and every non-window + cache take the `windowed==false` path; for `n_stream==1` the window view is sized `get_size()` + and passes `k_shift_src` directly — BYTE-IDENTICAL to the pre-0117 op (the `n_stream>1` + non-windowed view is unchanged too). Host-only, no CUDA DSO rebuild (the tail rope is an + existing ggml op dispatched to CPU). +- Markers: `opencoti-hook: rolling-kv-shift` — 2 sites (`get_can_shift`, `build_graph_shift`), + RENAMED from 0115's single-site `rolling-kv-shift-guard`. The 0115 patch file and its historic + marker text remain (0117 supersedes its behavior additively; 0115 stays in the chain). +- VALIDATED (#639, bs2 GPU0, clean/no-contention, Qwen2.5-14B-1M-Q8_0, `-c 12288 --context-shift + --keep 256`): spilled (`--vram-target 18000`, 6912 resident / 5376 host tail = 1008 MiB) → + shift ENABLED, coherent through the shift (ZULU-9999 present, monotonic run 1219, output + byte-identical to the fully-resident control), NO assert. Decode-tps == pre-#639 baseline + within noise (74.88→43.67 over ~2800 tok vs baseline 74.47→40.25); the re-rope is shift-event + only, no per-token cost. The 26.9-tps "regression" seen in the first gate was GPU/host + contention (bug-2149), not this patch. Byte-identical `git apply` of 0117 == the live edit. + +Chain state after 0117: 63 patches. + +## 2026-07-11 — 0118 T\*-aware auto rolling-KV spill (HAL #582 P1+, #621 follow-on) + +- `0118-kv-auto-tier-tstar-spill.patch` (#621 follow-on) — extends the 0116 auto-tier helper + `opencoti_auto_select_kv_tier` so it can KEEP a higher-quality tier (up to f16) with a small + host-tail spill when that spill is provably cheap, instead of always quantizing to the first + fully-resident rung. The ladder fit-condition changes from "first tier fully resident" to + "first tier whose SPILL ≤ `T*_bytes`", `T*_bytes = (target_drop·t_resident − c_engage)·bw_eff` + (target_drop default 0.20, env `OPENCOTI_KV_TSTAR_DROP`; clamp ≥0, ≤ cliff env + `OPENCOTI_KV_TSTAR_MAX_SPILL_MIB` default 800). Boot-measured terms: T1 explicit-axis handling; + T2 `bw_eff = min(pcie, host_ram)` with a NEW host 1-thread memcpy probe (`OPENCOTI_KV_HOST_RAM_GBPS`); + T3 `fa_ms_cell`+`c_engage` from the reused FA compute microbench (#351, `OPENCOTI_KV_CENGAGE_MS`); + T4 the drop-model. Fast-path guard keeps the "f16 fits" boot byte-identical; probe-unavailable or + T*≤0 degrades exactly to 0116's fully-resident gate; the ctx-value gate (req #2) falls out of the + formula (short ctx ⇒ tiny t_resident ⇒ c_engage dominates ⇒ quantize). +- Strictly opt-in (`OPENCOTI_KV_AUTO_TIER=1`, the same 0116 switch) and a byte-identical no-op when + unset. HOST-ONLY: the FA microbench dispatches to the existing CUDA DSO — no ggml-cuda.so rebuild. +- Marker: `opencoti-hook: kv-auto-tier` — EXTENDS the existing 0116 site (the ctor helper + + its one call), no new hook site. The 0116 marker text covers it. +- VALIDATED both hosts (T2-T4 terms fire; solidPC PCIe-bound bw_eff 6.7, bs2 host-RAM-bound 9.3-10.3; + drop-model monotone in target_drop). T5 RULER-niah quality/tps gate PASS (bs2 GPU0, + Qwen2.5-14B-1M-Q8_0, `-c 65536`, `--vram-target 27000` → ~2 GB f16 spill): auto → q8_0/q8_0 + (T* only ~14 MiB), f16-window-spill decode-tps drops 78.5% (≫ 20% target), q8 niah == f16 niah + == 100 (lossless), q8 tps 35.3 = 3.3× f16-spill 10.8 — quantize verdict correct, fallback + dominates. Byte-identical `git apply` of the full chain 0001..0118 == the live edit (cmp OK). + +Chain state after 0118: 64 patches. + +### `0119-p0-dcaon-scalar-inregister` (#582-P0 / #643 + #644) + +Scalar q4_0/q4_1/q5_0/q5_1/q6_0/q8_0 K/V read IN-REGISTER inside the multichunk +`flash_attn_ext_f16_dca_fused` MMA kernel, killing the whole-cache `to_fp16_nc` +dequant-lift that dominated quant-KV DCA-on decode (HAL §3b profile ~29.5 s/decode). +Scalar generalization of #629's turbo-V in-register read; reuses the SAME +`load_tile_dequant` / `get_dequantize_V` path (no new kernel math). + +- Marker: `opencoti-hook: P0 DCA-on scalar in-register` — 3 sites: `fattn-common.cuh` + (the 4 new `ne==8` V-reader branches, q4_1/q5_0/q5_1/q6_0 — the only correctness + surface, bug-2152; the MMA loader stays fixed at `ne_per_chunk=8`, SHARED with the + proven turbo path, NOT changed); `fattn-mma-f16.cuh` (skip in-launch K materialize + for a scalar `type_K`, mirroring the existing `type_V==F16` guard; + D128 scalar + externs); `fattn.cu` (decode-gated dispatch, `Q->ne[1] ≤ WS2_NQ_MAX`=16, env + `WS2_DCA_SCALAR_KV` default on — matched pair dispatches its typed instance, any + other combo falls through to the shipped F16-materialize default). +- 40 NEW template-instance TUs (`fattn-mma-f16-dca-fused-scalar--d128-{0..3}.cu`; + 4 ncols2 shards × 10 pairs) — named `fattn-mma-*.cu` so `collect_gpu_sources` globs + them. D128 only (Qwen full-attention); D256/D512 scalar deferred (Gemma KV is turbo, + already in-register). +- Correctness: #643 beachhead q8_0-K/q4_0-V @D128 ctx128k `real_frac 0.0` + 2.13× + decode (14.72→31.33 tps); #644 all 10 pairs @D128 32k multichunk DCA-on `real_frac + 0.0`. Byte-identical `git apply` of 0119 onto a reconstructed 0118 baseline == the + live edit (cmp OK, 43 files). Additive-only, touches only nested llama.cpp files (no + outer-file conflicts). DSO 2.5→3.83 GB, absorbed by the fatbin-last fragment + (bug-2140); `-mcmodel=large` stays OFF. **Applies after `0118`.** + +Chain state after 0119: 65 patches. + +### `0120-p2-mixed-kv` (#622 / #582-P2) + +Position-axis **mixed KV** for rolling-KV spill: the resident window `[0,wc)` keeps +the boot KV type (e.g. `q8_0`) while the spilled host tail `[wc,n_kv)` may carry a +MORE-compressed type (e.g. `q4_0`). Recent (attention-dominant) tokens stay +high-fidelity on-device; the older tail pays fewer H2D bytes per decode step. +Position-axis complement to the head-axis (#620/#643 P0) and boot-tier (#621 P1) +residency levers. New CLI `-ctkt`/`-ctvt` set the tail K/V type; sentinel +`GGML_TYPE_COUNT` (flag unset) means "tail == window" → every path is byte-identical +to the shipped uniform-window behaviour. + +- Marker: `opencoti-hook: P2 mixed-kv` — additive 5-point plumbing carrying the tail + type from CLI to the pinned host allocation: `common/arg.cpp` + (`--cache-type-k-tail`/`-v-tail` → `common_params.cache_type_k_tail`/`_v_tail`, + `common/common.{h,cpp}`) → `llama_context_params` (`include/llama.h`) → + `llama_cparams` (`src/llama-cparams.h`) → kv-cache ctor + (`src/llama-kv-cache.{cpp,h}`, `src/llama-kv-cache-iswa.{cpp,h}`, + `src/llama-context.cpp`, `src/llama-model.cpp`), which allocates + `k_cpu_per_stream`/`v_cpu_per_stream` at the tail type. **Decode** (the win): the + POSITION_WINDOW streaming FA op (`ggml/src/ggml-cuda/fattn.cu`) already reads + window + tail each in-register at its own type (reuses #620/#629 readers) — no + whole-cache f16 materialise. +- Two correctness-fallback fixes in `src/llama-kv-cache.cpp`, both host-only, both + byte-identical when tail type == window type: + - **bug-2161** — the get_k/get_v concat-reassembly accessor (prefill / graph-reserve, + n_q > `WS2_NQ_MAX`) feeds `ggml_concat`, which asserts `a->type == b->type`. Fixed + with a file-local `poswin_lift_to_f16` (mirror of `dca_lift_to_f16`: f16 + passthrough; scalar quants via `ggml_cast`→F32→F16, byte-exact GPU CPY) applied to + both region views only when the two types differ. + - **bug-2162** — the Stage 3c-6 state-IO window paths (K/V × write/read) used the + window row size for the tail region, so an in-memory context-checkpoint + (llama.cpp PR16391, on by default) asserted on the 2nd prompt at a mixed config. + Fixed with a per-region `ggml_row_size(k_cpu->type / v_cpu->type, n_embd_*_gqa)`, + symmetric across writer and reader; parity assert recomputed from both region sizes. +- Gate (S7, bs2 Qwen2.5-14B-Instruct-1M-Q8_0, `-c 24576` `--vram-target 18000`, + logit-equiv vs full-resident gold; NEVER greedy needle): **correctness** `real_frac + 0.0` for U8/MIX/U4, advisory mean-TV **U8 0.0069 < MIX 0.0110 < U4 0.0118** (mixed + strictly between uniform-q8 and uniform-q4, closer to gold than uniform-q4); + **throughput** MIX 28.1 vs U8 26.4 tps = **+6.4%** (host tail 11264 cells); + **regression** `-ctkt q8_0 -ctvt q8_0` sentinel byte-identical to unset (REG==U8). + The gate exercises context checkpoints, so it doubles as a checkpoint round-trip test + for bug-2162. Byte-identical `git apply` of 0120 onto a reconstructed 0119 baseline == + the live edit (0 files differ) + clean reverse-apply on the live tree. Additive-only, + touches only nested llama.cpp files. **Applies after `0119`.** + +Chain state after 0120: 66 patches. + +### `0121-p2-auto-tail` (#653 / #582-P2) + +**P2-auto boot policy**: when the P1 boot auto-tier (`opencoti_auto_select_kv_tier`, +0116/0118) is FORCED to spill — the auto-selected KV window still overflows the VRAM +budget, so the rolling-KV path spills a tail regardless — auto-pick a MORE-compressed +type for the spilled tail `[wc,n_kv)` than the resident window, reclaiming H2D bytes +on the oldest (least-attended) cells for free. Reuses the 0120 tail plumbing +(`type_k_tail`/`type_v_tail`, `GGML_TYPE_COUNT` sentinel = "tail == window"); this +patch only adds the boot POLICY that fills those sentinels in. + +- **Opt-in is SEPARATE from P1**: `OPENCOTI_KV_AUTO_TIER_TAIL=1` (in addition to + `OPENCOTI_KV_AUTO_TIER=1`). With P1 alone the tail stays == window, so a P1-only + boot is byte-identical to before this patch — the compressed tail engages only when + the operator also asks for it. An explicit `-ctkt`/`-ctvt` (tail != COUNT) is honored + and never overridden, mirroring how `-ctk`/`-ctv` block the window auto-tier. +- **Policy — "q4_0 floor when window > q4_0"**: for each axis independently, if the + chosen window type is higher-bit than q4_0 (`f32/f16/bf16/q8_0/q6_0/q5_1/q5_0`) and + that axis's tail is unset, set the tail to q4_0; otherwise leave tail == window (so a + window already at/below q4_0 is not "upgraded" to a coarser tail). The policy hook + (`maybe_auto_tail`) fires ONLY on the two spill exits — the "window kept but spills" + T\* branch (chosen==0) and the "chosen tier still spills" tail — so it never touches + the fully-resident fast path (early return). +- Marker: reuses the existing 0116 `opencoti-hook: kv-auto-tier` site — this edit lives + inside `opencoti_auto_select_kv_tier` and its one call site, exactly like 0118. No new + inline hook site; the 0116 marker text covers it. +- Gate (bs2, Qwen2.5-14B-Instruct-1M-Q8_0, `-c 24576` `--vram-target 18000`, + `--flash-attn on`, logit-equiv vs full-resident gold; NEVER greedy needle): **P2AUTO** + (both envs) boot log shows `auto KV tail = q4_0/q4_0` over a real ~189 MiB / 3072-cell + spilled region (window q5_1/q4_0 at the second-pass binding budget — bug-1342 two-pass + compute reserve runs the auto-tier twice, the first looser pass fits, the binding pass + spills), `real_frac 0.0`, advisory mean-TV 0.0087 (not worse than the uniform-tail + P1AUTO floor). **P1AUTO** (`OPENCOTI_KV_AUTO_TIER=1` only) shows NO tail line — + regression shield: byte-identical to the shipped P1 boot. HOST-ONLY (`src/llama-kv-cache.cpp` + is libllama in the host binary, not the CUDA DSO): `make` rebuild, DSO ccache-hits. +- Byte-identical `git apply` of 0121 onto a reconstructed 0120 baseline == the live edit + (cmp OK) + clean reverse-apply. Full chain 0001..0121 strict-applies from clean 0.10.3. + Additive-only, touches only the nested `llama.cpp/src/llama-kv-cache.cpp`. **Applies + after `0120`.** + +Chain state after 0121: 67 patches. + +## 2026-07-13 — 0122 RYS layer duplication `--repeat-layers` (#656–#668) + +### `0122-rys-layer-duplication` (RYS-1..12) + +**`--repeat-layers SPEC`** — load-time, weight-shared re-run of source transformer +layers *in place* (RYS / dnhkng "passthrough self-merge" — mergekit passthrough + +SOLAR depth-upscaling, done at inference time with no new weights). SPEC = +`;`-separated `i,j` blocks re-running layers `[i,j)` a second time right after the +original run; the **empty** plan ⇒ off ⇒ **byte-identical** to before this patch (eff +== src, identity map). All host-side; **no CUDA**. + +- **Data plane.** `common_params.repeat_layers` (parsed in `common/arg.cpp`) → + `cparams.layer_plan`, a `std::vector` of length `n_layer_eff` where + `layer_plan[eff] = src` (the original layer each effective step re-runs). Parsed + + validated in `src/llama-context.cpp` (`opencoti_parse_repeat_layers`): fail-loud on + out-of-range or malformed spec. The forward graph iterates the EFFECTIVE plan and + passes `eff` (the KV slot) to `build_attn`; **weights/hparams are keyed by + `src = layer_plan[eff]`** (RYS off ⇒ eff==src ⇒ every existing lookup is identity). +- **KV allocation across all cache families.** `map_layer_ids` + slot allocation + extended to `n_layer_eff` in: the **unified** cache (`llama-kv-cache.{cpp,h}`), the + **iSWA dual-cache** (`llama-kv-cache-iswa.{cpp,h}` — base + swa each eff-length, the + `is_swa` filter resolved via an inner eff→src so a duplicated global stays global), + the **recurrent/hybrid** state caches (`llama-memory-recurrent.{cpp,h}`, + `llama-memory-hybrid{,-iswa}.{cpp,h}`), and the **DCA** build path (`dca.cpp` — eff→src + for the single per-source `is_swa` hparam it reads). +- **Per-arch forward loops** (`src/models/*.cpp`): Qwen2/2.5/3 dense, Qwen2-MoE + + Qwen3-MoE, Qwen3.5-MoE + Qwen3-Next recurrent (gated-delta-net), and Gemma-4 iSWA + dense (12B/31B, per-layer `√` embed-scale keyed by src) + A4B (`gemma4.cpp`). Bucket-C + recurrent boot crash root-caused + fixed here (auto-FA/GDN device resolvers parsed the + EFFECTIVE idx from the op name → `dev_layer(eff)` out-of-range; eff→src map at 3 sites, + bug-2160). +- **Boundary-layer advisory** (bug-2164): a boot `LLAMA_LOG_WARN` when the plan + duplicates a layer within the first/last `max(3, n_layer/8)`. Duplicating early/late + layers commonly yields incoherent output — this is a **MODEL** property (franken-merge + boundary fragility), **NOT an engine bug**: verified by a middle/boundary × SWA/global + 2×2 on native Gemma-4-A4B via the CHAT endpoint (middle SWA L4/L10 + middle GLOBAL + coherent; boundary GLOBAL L29 garbage — garbage tracks boundary-ness, independent of + SWA/global). Coherent band ≈ L4..L24 (Gemma-4-A4B), only the last layer on Qwen3-8B. + The advisory is the entire "fix"; there is **no correctness code change** — the engine + is provably correct on middle layers. +- **Markers**: `opencoti-hook: rys-layer-dup` (vendored-source markers only — additive + vendored edits registered here; do **not** change the `git grep "opencoti-hook:" + packages/ docs/` upstream-opencode inventory). Provenance: opencoti-original (adopts + dnhkng RYS-II + mergekit-passthrough conventions). +- **Gates** — coherence smoke via the **CHAT** endpoint (RYS *intentionally* changes the + model, so logit-equiv-vs-baseline is the wrong bar; and the target models are + thinking-models whose answer lives in `reasoning_content` — raw greedy `/completion` + is garbage even at baseline). Qwen3-8B dense off/plain/DCA/vram-window/sparse all + coherent; Gemma-4 A4B (MoE-iSWA) + 31B (dense-iSWA) middle-layer dup coherent, boundary + dup garbage + advisory fires; RYS × rolling-KV residency window × DCA-on composes + coherently on iSWA (window engages, no boot crash); recurrent (qwen35moe GDN) boots + + self-stacks including full-model `0,40`. +- Byte-identical `git apply` of 0122 onto a reconstructed 0121 baseline == the live edit; + full chain 0001..0122 strict-applies from clean 0.10.3. TS adapter: `repeatLayers` + field/flag in `packages/opencoti-llamafile/src/{config,launch}.ts`. **Applies after + `0121`.** See [features/rys_layer_duplication.md](../features/rys_layer_duplication.md). + +Chain state after 0122: 68 patches. + +## 2026-07-13 — 0123 RYS template probe `--rys-probe` (#669–#670) + +### `0123-rys-probe` (rys-probe v1.1) + +**`--rys-probe`** — a new host-only PROGRAM MODE (`ProgramMode::PROBE`) that maps a +GGUF for good RYS layer-duplication templates (companion to the 0122 `--repeat-layers` +engine), the cheap alternative to dnhkng RYS-II's days-long beam + XGBoost-surrogate + +graded-benchmark search. + +- **New standalone file** `llamafile/rys_probe.cpp` (driver, `lf::rys_probe_main`), detected + in `llamafile/args.cpp` (→ `ProgramMode::PROBE`, enum in `args.h`), dispatched from + `llamafile/main.cpp`, built via `llamafile/BUILD.mk`. Four additive `common_params` fields + (`rys_probe`, `rys_probe_widths="auto"`, `rys_probe_band="auto"`, `rys_probe_topk=10`) + + four `--rys-probe*` flags in `llama.cpp/common/{common.h,arg.cpp}`. +- **Mechanism**: parse args as the PERPLEXITY example, disable `-fit` (its trial context + aborted the flash-attn print), `common_init_from_params(model_only=true)` — load ONCE. + For each contiguous `[i,j)` in the band `[max(3,n/8) .. n-max(3,n/8))` × widths, rebuild + ONLY the `llama_context` (`llama_init_from_model`, `cp.repeat_layers="i,j"`; layer_plan has + no runtime setter, weights are plan-independent), score **ΔPPL** (small corpus, compact + batched NLL) + a **graded 24-item task-probe** (greedy + substring early-stop), free ctx. +- **Output**: ranked table + **exactly two templates** — MOST EFFICIENT (max ΔPPL/added-layer) + + MAX GAIN (lowest ppl); honest "same block is both" when no wider block wins. The `task` + score is both a rank signal and a guardrail (drop candidates below base `task`). The two + templates feed `perf/llamafile/rys-probe-eval.sh` for the real downstream verdict. +- **Markers**: none needed — this is a NEW standalone mode, not an invisible behavior change + to an upstream file; a run without `--rys-probe` is byte-identical (the model path is + untouched). Provenance: opencoti-original (cheap analogue of dnhkng RYS-II + mergekit + passthrough conventions). No TS adapter (a diagnostic CLI, not a runtime server flag). +- **Gates**: host builds clean (incremental; the `common_params` field-ADD needs a clean + `rm -rf o` rebuild — bug-2160/2166 ABI-skew class — but the later widths default-VALUE + change is ABI-safe). Smoke on Qwen3-4B-Q4_K_M (band [10,16) auto, 14 cand): base ppl 30.15 + task 0.917 → winner 11,13 (ppl 28.03, task 0.958). Mapped Qwen3.6-35B-A3B-UD-IQ3_XXS + (n_layer 40, band [5,35), 109 cand) on the 3090. +- Byte-identical `git apply --reverse --check` of 0123 against the live tree (hunks match + exactly); `common.h`/`arg.cpp` baseline reconstructed from the post-0122 pre-rys-probe + vendor-backup, the 5 outer `llamafile/*` files diffed against HEAD (== post-0122; no prior + patch touches them). Full-chain 0001..0123 strict-apply from clean 0.10.3 = Phase B (post-map, + via `build:llamafile:check`). **Applies after `0122`.** See + [features/rys_probe.md](../features/rys_probe.md). + +Chain state after 0123: 69 patches. + +## 2026-07-13 — 0124 RYS×MTP draft-context guard (bug-2171) + +### `0124-rys-mtp-ctx-guard` (bug-2171) + +**RYS `--repeat-layers` must not propagate into an MTP/NextN draft context.** With RYS +active *and* self-speculation on (`--spec-type draft-mtp`, Qwen NextN or the gemma4 +assistant), the draft `llama_context` inherited the target's `params.repeat_layers` (it +is built from `common_context_params_to_llama(params_base)`), so its `cparams.layer_plan` +was RYS-duplicated too. But the NextN head sits at the canonical `il = n_transformer` +(`n_layer - nextn_predict_layers`), which the RYS eff→src map never contains → boot aborts +at draft-context init with `cpy_k: map_layer_ids MISS il=n_transformer (n_keys=0)` → +`failed to create MTP context`. RYS is a **target-only** inference capability: in self-spec +the NextN head consumes the target's *already-RYS* hidden state through shared memory, so +the drafter must run the plain base layer stack. + +- **Fix** (`src/llama-context.cpp`, in the RYS plan-expansion block of the ctor): gate the + spec on context type — + `const char * rys_spec = (cparams.ctx_type == LLAMA_CONTEXT_TYPE_MTP) ? nullptr : params.repeat_layers;` + then parse that (nullptr ⇒ empty plan ⇒ eff==src identity for the draft). The **target** + context still receives `params.repeat_layers` unchanged, so RYS on the target is + untouched; only the draft context is forced plain. `sched_reserve()` mirrors the same + guard so the reserve graph matches. +- **Lossless**: greedy self-spec has the target verify every drafted token, so the + drafter running the plain stack (vs an RYS-duplicated one) changes only acceptance/tps, + never output. Boot now clean: RYS `5,9` + `draft-mtp` on Qwen3.6-35B-A3B boots with + `RYS … 44 effective layers` **and** `ctx_dft=yes` (no MISS). +- **Byte-identical when RYS is off** (empty `repeat_layers` ⇒ the guard picks nullptr + either way) and when MTP is off (target path unchanged). Marker: reuses the 0122 + `opencoti-hook: rys-layer-dup` site (same feature, no new hook). HOST-ONLY (no CUDA). + Provenance: opencoti-original. **Applies after `0123`.** + See [features/rys_layer_duplication.md](../features/rys_layer_duplication.md). + +Chain state after 0124: 70 patches. + +## 2026-07-14 — 0125 hybrid order-agnostic rolling-KV window (bug-2172 / #672) + +### `0125-hybrid-window-orderagnostic` (bug-2172 / #672) + +**Order-agnostic rolling-KV position-window scatter for a HYBRID model's attention +sub-cache.** On qwen35moe / qwen3next (gated-delta-net recurrent + full-attention + MoE + +NextN), the hybrid memory is `llama_memory_hybrid` wrapping a `llama_kv_cache` (`mem_attn`, +`!is_recurrent`) and a `llama_memory_recurrent` (`mem_recr`). When rolling-KV arms its +position-window on `mem_attn` under tight VRAM / large ctx, the recurrent batch pipeline +(`split_seq` + `[TAG_RECURRENT_ROLLBACK_SPLITS]` + `find_slot` cell-reuse/wrap) hands the +attention scatter **non-monotone destination cell idxs**, which trip the append-only window +scatter's monotone `GGML_ASSERT` in `cpy_k` (`llama-kv-cache.cpp:4083`, pre-patch). + +- **Fix** (host-only; single TU `src/llama-kv-cache.cpp` + a one-line setter in + `src/llama-kv-cache.h` + a one-line call in `src/llama-memory-hybrid.cpp`): a new + `llama_kv_cache::append_only` flag (default **true**) is set **false** only by + `llama_memory_hybrid` on its `mem_attn`. Every dense/iSWA/unified cache keeps + `append_only=true` → the append-only fast path is **byte-identical + unchanged**. When + false **and** in window mode: + 1. the resident window (`k/v_per_stream`) and host tail (`k/v_cpu_per_stream`) each get + **+1 "trash" sink cell** (`wc_alloc` / `tail_c_alloc`); the append-only path never + touches it (reads/writes only `[0,wc)` / `[0,tail_c)`), so its compute is byte-identical; + 2. `build_input_k/v_idxs` emit a **2-row `[n_tokens,2]`** routing tensor (row 0 = + window-route → dest cell or window-trash; row 1 = tail-route → dest−wc or tail-trash), + filled by `set_input_k/v_idxs`. `ne[0]` stays `n_tokens` so every `->ne[0]==n_tokens` + graph guard holds; + 3. `cpy_k` / `cpy_v` scatter the **full per-stream K/V into BOTH** window and tail via two + arbitrary-dest `ggml_set_rows` (non-sorted dests are legal → the monotone assert is + gone), off-target copies landing in the never-read trash cell. **Same op-count at reserve + and live** (exactly 2 set_rows/stream) so the reserve topology matches. +- **Read side UNCHANGED**: `get_k/v_window` reads `[0,min(n_kv,wc))`, `get_k/v_tail` reads + `[0,n_kv−wc)` — these already cover every active scattered cell, and the streaming-FA + `KQ_mask` `-INF`-masks the rollback holes by absolute key position. **Mask contract + verified**: `GGML_OP_STREAMING_FLASH_ATTN` is fully mask-driven, per-cell, + absolute-key-position-indexed, softmax-denominator-normalized → **zero CUDA change, no DSO + rebuild** (host `make` only). +- **Scope**: iSWA-hybrid is out of scope (no current model is both hybrid AND SWA; the window + never arms on `swa_type!=NONE`). Documented as such. +- **VALIDATED** (solidPC host `make`): qwen35moe auto-spill boots with NO assert + correct + output; dense (Qwen) + iSWA (Gemma) regression shields byte-identical (append_only path + untouched). Snapshot-diff capture (**3 files**) from a reconstructed post-0124 baseline; + **byte-identical-verified** (`git apply` onto 0124-state == live, all 3 files sha256-match). + Marker: `opencoti-hook: hybrid-window-orderagnostic` (**3 sites**: the `append_only` flag + + the `cpy_k` / `cpy_v` scatter branches). HOST-ONLY (no CUDA). Provenance: opencoti-original. + **Applies after `0124`.** See [features/rolling_kv.md](../features/rolling_kv.md). + +Chain state after 0125: 71 patches. + +### `0126-p0-dcaoff-scalar-mma-decode` (#582-P0 / #620, #644-mirror) + +**DCA-off scalar-quant MMA decode.** Single-token quant-KV decode with DCA OFF was stuck +on the FA-VEC reader, which HAL §3b measured as kernel-bound (~2× f16 regardless of bytes +moved — the read never became bandwidth-bound). f16 at high-GQA / long-KV already falls to +the tensor-core MMA path (the `fattn.cu` selector); this patch gives a scalar-quant K/V +cache the SAME route: it reads the raw quant blocks straight into the MMA smem tile +in-register (`flash_attn_ext_f16_load_tile_dequant` / `get_dequantize_V` — no whole-KV +`to_fp16_nc` materialise) and runs the tensor-core `mul_mat`. Quant decode therefore +becomes bandwidth-bound and BEATS f16 (fewer KV bytes moved). This is the **DCA-off +mirror** of `0119`'s DCA-on scalar in-register read; the two share the same in-register +reader family (#629/#630). + +- **--fa-all-quants premise**: opencoti's CUDA build passes `--fa-all-quants` + (`build-pipeline.ts` → `-DGGML_CUDA_FA_ALL_QUANTS`), so a native FA-VEC instance for + every scalar pair already EXISTS. #620 is an **MMA fast-path OVER** that VEC path, NOT + new support. The env `OPENCOTI_QUANT_MMA_DECODE` (default-on) selects it; env=0 reverts + each pair to its native FA-VEC path (graceful fall-through, never a boot refusal). Scalar + KV is not FWHT-rotated, so there is no `wht_o` / de-rotate (unlike turbo #630). +- **Matrix** (D128 served pairs, mirrors the #644 scalar set): hi-K/cheap-V + {q8K/{q4,q5,q6,q8}V, q6K/q4V} + symmetric diagonal {q4_0, q4_1, q5_0, q5_1, q6_0}. +- **Mechanism** (all additive; nested llama.cpp only): `fattn.cu` — 10 forward-decls + + `octi_quant_mma_decode_enabled()` (env gate) + the shared predicate + `octi_scalar_mma_pair_d128(K,V)` (the served D128 scalar pair set); an MMA-dispatch block + (`OCTI_SCALAR_MMA_D128` macro over the 10 pairs) in + `ggml_cuda_flash_attn_ext_mma_f16`; guard admittance + a VEC-selector escape in + `ggml_cuda_get_best_fattn_kernel`, both gated on `dst->src[7]==nullptr` (DCA off), so an + admitted pair falls straight to `BEST_FATTN_KERNEL_MMA_F16`. 10 new + `template-instances/fattn-mma-f16-scalar---d128.cu` typed instances, each with an + entry + ncols1/ncols2 gqa dispatch calling + `ggml_cuda_flash_attn_ext_mma_f16_case<128,128,…,GGML_TYPE_,GGML_TYPE_>`. + Auto-collected by `build-functions.sh`'s `fattn-mma-*.cu` glob (#531). +- **VALIDATED** (bs2 GPU1, Qwen2.5-14B-1M Q8_0, ctx 65536, n_kv≈44k, correctness via + logit-equiv `real_frac`, never greedy needle — bug-263/270): beachhead q8_0-K/q4_0-V + **70.45 tps vs f16 60.99 (1.15×) vs native VEC 30.60 (2.30× — the §3b tax fully killed)**, + `real_frac 0.0` vs BOTH f16 and native VEC (top-token identical, lossless). Generalization + (5-config @ ctx 65536): decode tps rises monotonically as V gets cheaper — **f16 62.15 < + q8q4 68.47 < q8q8 69.45 < q6q4 70.31 < q4q4 73.97**, all boot coherent, `real_frac 0.0` vs + f16. Snapshot-diff capture (**11 files**: 1 mod `fattn.cu` + 10 new TUs) from a + reconstructed post-0125 baseline (the pre-#620 `fattn.cu`, verified 4 hunks all + `octi_scalar_mma`); **byte-identical-verified** (`git apply` reverse-check on live PASS + + forward `base+0126` == live, all 11 files `cmp`-identical). Marker: `opencoti-hook: P0 + DCA-off scalar-quant MMA decode` (**3 sites** in `fattn.cu`: the predicate + MMA-dispatch + block; the guard admittance; the selector escape). CUDA-only (host binary unchanged). + Provenance: opencoti-original. No TS adapter (kernel-routing env, default-on; no + user-facing server flag — same as `0119`). See + [evaluations/hal_kv_compression.md](../evaluations/hal_kv_compression.md). **Applies after + `0125`.** + +Chain state after 0126: 72 patches. + +### `0127-p0-dcaoff-scalar-mma-decode-d256-d512` (#582-P0 / #674) + +**DCA-off scalar-quant MMA decode — D256/D512 generalization.** `0126` (#620) gave a +scalar-quant K/V cache the tensor-core MMA decode fast-path on **D128** (Qwen +full-attention). This patch generalizes that same fast-path to **Gemma-4's two head-dims**: +**D256** (SWA-local layers) and **D512** (global layers — the KV that grows with context, +where the HAL §3c/§3d −28…−35 % kernel-bound quant tax bit hardest). A single Gemma boot now +routes a scalar-KV decode through the tensor cores on **every** layer via the per-layer +head-dim predicate. Making Gemma scalar decode bandwidth-bound (so it beats f16) is what makes +scalar-quant KV the **preferred high-VRAM serving path over turbo** — turbo is slower *and* +decisively lower quality. + +- **Mechanism** (0126's, widened; all additive, nested llama.cpp only): the shared predicate + `octi_scalar_mma_pair_d128` → **`octi_scalar_mma_pair`** now admits symmetric head-dim + `K==V ∈ {128,256,512}` (was 128-only); the forward-decls become an + `OCTI_DECL_SCALAR_MMA(kk,vv)` macro emitting all three head-dims; the `OCTI_SCALAR_MMA_D128` + dispatch macro → **`OCTI_SCALAR_MMA(kk,vv,base)`** routing on `d ∈ {128,256,512}` to + `base##_d{128,256,512}`. The two consult-sites (the `--fa-all-quants` dead-code guard + admittance + the selector escape, both `dst->src[7]==nullptr`) already read only that + predicate, so they generalize for free. **20 new** + `template-instances/fattn-mma-f16-scalar---d{256,512}.cu` (D256/D512 twins of the + 0126 D128 TUs; `ggml_cuda_flash_attn_ext_mma_f16_case<256,256,…>` / `<512,512,…>`). D256/D512 + MMA + in-register scalar V-readers (`ne==8`) are proven by the pre-existing turbo D256/D512 + instances (#630) and f16's `switch_ncols2<256,256>`/`<512,512>`. Auto-globbed (#531). +- **VALIDATED** (bs2 GPU1, Gemma-4 A4B native google 26B-A4B-128e Q4_K_M, **DCA-off**, + correctness via logit-equiv `real_frac` vs f16 — Gemma A4B PPL/KLD is garbage, so + correctness is top-token identity; never greedy needle): + - **ctx 64k** (nkv 42.5k): f16 162.13 < q4q4 170.17 < q8q8 173.86 < **q8q4 174.94** tps + (q8q4 **+7.9 %** over f16; env=0 VEC fallback **122.17** = the §3c kernel-bound tax → MMA + is **+43 %** over VEC, tax fully erased). `real_frac 0.0` for q8q4/q8q8/q4q4 (lossless). + - **ctx 128k** (nkv 112k): f16 152.15 < q4q4 152.95 < **q8q4 154.46** tps (q8q4 **+1.5 %**; + §3c had q8q4-VEC at **−35 %** here). `real_frac 0.0` for q8q4/q4q4 (lossless at depth). + Win narrows deep as MoE-FFN + per-step launch overhead dilute the KV-bandwidth delta, but + scalar stays ≥ f16 and top-token-lossless at both depths. +- Snapshot-diff capture (**21 files**: 1 mod `fattn.cu` + 20 new TUs) from the 0126-level + `fattn.cu` extracted byte-exact from the `20260714-042251-pre-0126-620-mma-capture` + vendor-backup tarball (verified 19 D128 refs; diff-vs-live = exactly the #674 generalize); + **byte-identical-verified** (`base+0127` == live, all 21 files `cmp`-identical). Marker: + `opencoti-hook: #582-P0 DCA-off quant-MMA decode #620 D128 + #674 D256/D512`. CUDA-only. + Provenance: opencoti-original. No TS adapter (kernel-routing env, default-on; same as + `0119`/`0126`). See + [evaluations/hal_kv_compression.md](../evaluations/hal_kv_compression.md). **Applies after + `0126`.** +- **Build note**: the 20 new (esp. D512) instances grow `ggml-cuda.so` past 4 GiB (~4.43 GiB), + which tripped a latent build-pipeline bug (**bug-2178**): `sha256OfFile` read the whole DSO + into one `ArrayBuffer` and aborted (V8 max typed-array length), skipping the side-load + mirror. Fixed separately in `packages/opencoti-llamafile/src/build-pipeline.ts` (stream the + hash — O(1) memory). The DSO link itself holds under the #629 `-mcmodel=large` mitigations. + +Chain state after 0127: 73 patches. + +### 2026-07-14 — 0128 MTP spec-cycle host parity (bug-858 / #675) + +- `// opencoti-hook: mtp-verify-mmvf (bug-858/#675)` (`ggml-cuda/mmvf.cu`, 2 sites) — + TinyBLAS-gated `src0_small && ne11 <= 8` lift in the **f16 + bf16** ampere_mma branches of + `ggml_cuda_should_use_mmvf` (siblings of `0093`'s f32 site, bug-2103/#609). Upstream caps + these at `ne11==1` because its cuBLAS fallback wins past that; TinyBLAS builds have no + cuBLAS, so MTP verify batches (ne11 = n_accepted+1 = 2..4) dropped the thin bf16 + shared-expert gate GEMVs (`ffn_gate_inp_shexp`) onto a single-block tinyblasGE cliff. + Base decode (ne11==1) and cuBLAS builds unchanged. Patch `0128`. +- `// opencoti-hook: spec-clone-cheap (bug-858/#675)` (`common/sampling.cpp`) — + `common_sampler_clone` clones with empty `cur`/`cur_p` instead of deep-copying the + n_vocab-sized (~1.8 MiB) scratch every speculative cycle: under the cosmo dlmalloc runtime + that copy is a fresh mmap + fault + munmap = 1.071 ms/cycle (the entire measured spec + wall-clock gap vs b9859, whose glibc reuses the block). `cur` is write-only scratch + (`set_logits()` repopulates before any read); the copied `cur_p.data` aliased the source + sampler's buffer anyway. Output bit-identical on greedy; clone 1.071 → 0.005 ms/cycle. + **Sync note**: on future bumps, re-check that upstream hasn't started reading a clone's + `cur`/`cur_p` before `set_logits()` — if it has, restore the copy (and pool it instead). + Patch `0128`. +- Gate: Qwen3.6-35B NextN n3 282.6 vs b9859 287.4 (−1.7%, was −12%); base parity; accept + bit-identical. Full matrix + 9-prompt bench: + [evaluations/three-way-tps.md](../evaluations/three-way-tps.md) "2026-07-14 refresh". + +Chain state after 0128: 74 patches. + +### 2026-07-14 — 0129 rys-probe nested capture + 0130 parallel≥2 boot/checkpoint robustness (bug-2182/2183) + +- `0129-rys-probe-nested-args` — no new markers; completes the `0123`/#669 rys-probe capture: + the nested `common/common.h` `common_params` fields (`rys_probe*`) and `common/arg.cpp` + `--rys-probe*` `add_opt` registrations were live in the tree but missing from the chain + (a fresh apply would fail to compile the outer `rys_probe.cpp`). Chain-integrity only; + found by the 0130 per-file replay (baseline ≠ live on common.h). The rys hunks sit inside + regions already marked by `0123`'s rys-probe comments. +- `0130` bug-2182 (`src/llama-kv-cache.cpp`) — static `tie_scalar_f32()` helper + all 20 + dep-tie accumulator sites in the opencoti cpy_k/cpy_v scatter (M2 per-stream, hybrid + order-agnostic acc2, append-only windowed accumulate/emit) now collapse every tie operand + to exactly one F32 element (view ONE FULL BLOCK → cast F32 → view 1 element). Previously + the operand ne was the SOURCE blck_size (32 for q8_0/turbo) → `add([1],[32])` failed + `ggml_can_repeat` at `-fit`/graph-build under quant KV × parallel≥2 non-unified. All + inside opencoti-owned scatter code (no new upstream-facing marker; the enclosing blocks + carry the existing per-stream/rolling-KV hooks). f16/f32 KV graphs byte-identical. +- `0130` bug-2183 (`common/common.h`, `common/common.cpp`, + `tools/server/server-context.cpp`) — `common_prompt_checkpoint::load_tgt/load_dft` + return `bool` instead of GGML_ABORTing when `llama_state_seq_set_data_ext` returns 0 + (legitimate under `--kv-unified --parallel≥2` with another slot mid-decode); the server + task-launch restore site wipes the seq (tgt+dft) and falls back to full re-prefill via + the existing `do_reset` path; the speculative rollback/draft sites keep fail-fast aborts. + Comment-tagged `opencoti bug-2183` at all sites. **Sync note**: upstream declares these + loaders `void` — on future bumps re-apply the bool signature + the server fallback + branch together (they are one unit; a void loader with the fallback branch dead-codes + the recovery, an aborting loader with the branch reintroduces the crash). +- Gate: 3090, exact bs2 crash configs — q8 KV × `--parallel 2` non-unified × `-fit on` + boots n_slots=2 + decodes; forced cell-allocation failure under kv-unified parallel-2 + + prompt-cache → 0 aborts, FAILED→re-process fallback served all requests. + +Chain state after 0130: 76 patches. + +## 2026-07-15 — 0131-0133 introspection + version string + chain repair (c3) + +### chain repair (bug-2188) +A faithful full replay (clean 0.10.3 archive + Mozilla `llama.cpp.patches` overlay + +strict `git apply` 0001..0130) broke at `0124`: its second hunk duplicate-added the +Mozilla CPU auto-FA block already present from the overlay (the original capture used +an ad-hoc "reconstructed 0123 baseline" that lacked it). The redundant hunk was +dropped from `0124` (functional RYS guard hunk untouched). The same replay also +surfaced `llamafile/rys_probe.cpp` v2 (#673) shipped live but never captured — now +`0131`. **Whole chain 0001..0133 strict-applies from clean 0.10.3 and reproduces the +live tree byte-identically (DIFF_LINES=0).** Rule (bug-2108/bug-2188): capture +baselines come from `.opencoti/s5-build-0078-replay.sh`, never ad-hoc reconstructions. + +### `0131-rys-probe-task-battery-v2` (#673) +`llamafile/rys_probe.cpp` — Δtask-ranked balanced battery (see patches README row). +opencoti-owned file (introduced by 0123); no new upstream hook. + +### `0132-runtime-introspection` (#676/#677) +Markers: `// opencoti #676` / `// opencoti #677` comments at every touched site. +- `llama.cpp/include/llama.h` — `struct llama_opencoti_kv_info` + `llama_memory_opencoti_kv_info()` (additive API, after `llama_memory_can_shift`). +- `llama.cpp/src/llama-memory.h` — default-false virtual `opencoti_kv_info()` on `llama_memory_i`. +- `llama.cpp/src/llama-kv-cache.{h,cpp}` — impl reading live tensor types + `window_cells` residency. +- `llama.cpp/src/llama-kv-cache-iswa.{h,cpp}` — forward to non-SWA base, `is_iswa=true`. +- `llama.cpp/src/llama-memory-hybrid.{h,cpp}` — forward to `mem_attn`. +- `llama.cpp/src/llama-context.cpp` — C wrapper. +- `llama.cpp/tools/server/server-context.h` — `server_context_meta.json_opencoti` (sleep-safe /props). +- `llama.cpp/tools/server/server-context.cpp` — `get_opencoti_props()`, `get_meta()` init, `/props` key, `slot::to_json` opencoti block, lifetime draft counters (bug-2187). +- `llamafile/BUILD.mk` — bug-2186: `server.cpp.o` rule now depends on `tools/server/*.h` (stale-object ABI-skew guard). +Sync note: on a future llamafile bump, re-anchor the llama.h API block and the +server-context meta field/aggregate-init position; the virtuals are additive. + +### `0133-opencoti-version-string` (#613 / c3) +- `llamafile/version.h` — `OPENCOTI_TAG` default + `OPENCOTI_VERSION_STRING` (bump the default together with `vendors/llamafile/OPENCOTI_TAG` on every packaged cut). +- `llamafile/main.cpp` — `--version` + help banner print `llamafile vX.Y.Z (opencoti-X.Y.Z-)`; `llamafile vX.Y.Z` token must stay FIRST (scripts/llamafile-version.sh contract). +- `llamafile/BUILD.mk` — `OPENCOTI_TAG_FLAG` injection (make var from build pipeline) + `version.h` dep on `main.o`.