USAGE.md

# opencoti-llamafile — usage guide

How this engine diverges from upstream
[Mozilla-Ocho llamafile](https://github.com/Mozilla-Ocho/llamafile),
what the added features are, how each is gated, its knobs and
defaults, its limitations, and which features are meant to be used
together.

Audience: anyone running the packaged
`opencoti-llamafile-<ver>-<tag>-<arch>.llamafile` artifact as a local
inference server. Deep-dive design docs live in
[`docs/features/`](features/), measured evaluations in
[`docs/evaluations/`](evaluations/).

---

## Supported / target model families — read this first

opencoti-llamafile loads **any GGUF that upstream llama.cpp 
loads** — that part is inherited unchanged. But the opencoti feature
set (KV tiers, rolling-KV, DCA, MTP, sparse attention, RYS) is
developed, tuned, and correctness-gated on **two model families**, in
a deliberate primary/secondary split:

### Gemma-4 — PRIMARY target

| Model | Kind | Notes |
|---|---|---|
| Gemma-4 26B-A4B-128e ("A4B") | MoE | the flagship serving target; MTP-validated with its native gemma4-assistant drafter |
| Gemma-4 12B / 31B | dense | full feature validation incl. RYS + DCA |
| Gemma-4 E2B / E4B | elastic (E-series) | shared-KV elastic layers supported; MTP drafters available |

Gemma-4 is what the engine is *for*: its unusual head dims (256 and
512), the iSWA sliding/global dual KV cache, and the per-size
[gemma4-assistant MTP drafters](https://huggingface.co/ManniX-ITA)
all have dedicated kernels and graph paths here that upstream lacks
or handles slowly. `--spec-type draft-assistant`, D256/D512 FA-VEC +
scalar-MMA decode, iSWA-aware rolling-KV/SharedKVPool/DCA wiring are
all Gemma-4-first features.

### Qwen — SECONDARY target / verification family

| Model | Kind | Notes |
|---|---|---|
| Qwen3.5 / Qwen3.6 (e.g. 35B-A3B) | dense / MoE / hybrid (gated-delta-net) | NextN self-spec MTP (`--spec-type draft-mtp`, no external drafter needed) |
| Qwen2.5-14B-1M | dense, native-1M | the long-context/DCA validation vehicle |

Qwen is the standard-architecture (head_dim 128) counterweight:
every feature that ships is verified on it too, and it carries one
feature Gemma doesn't — **NextN self-speculation** (the model's own
MTP head drafts; fused multi-step, at/above upstream parity).

### Everything else

Other architectures run with upstream behavior and safe fallbacks,
but opencoti features are **unvalidated** there, and some are
arch-gated: MTP needs NextN tensors (Qwen-style) or a
gemma4-assistant drafter; RYS `--repeat-layers` supports the
qwen2/qwen3(+MoE)/qwen3.5/qwen3next/gemma-4 forward loops; DCA is
validated on Gemma-4 and Qwen2.5-1M. Quality gates (KLD,
RULER-niah) were run on the two families above — re-gate before
trusting aggressive KV tiers on anything else.

---

## 1. Relationship to upstream llamafile

opencoti-llamafile is **upstream llamafile 0.10.3 plus an additive
patch series** (`patches/` in the HF repo,
`vendors/patches/llamafile/` in the git repo — ~79 patches, numbered
`0006`–`0133`). Three properties are contractual:

1. **Off means off.** Every opencoti feature is opt-in behind a flag,
   env var, or per-request JSON field. With no opencoti flags set, the
   engine's compute path is **byte-identical to upstream** — this is a
   regression gate on every patch, not an aspiration.
2. **Lossless by proof, not vibes.** Features that touch the forward
   pass are gated by logit-equivalence / KLD / RULER-retrieval
   against vanilla, never by "the output looks fine". Speculative
   decode is verified-lossless (the output *is* the target model's).
3. **Single file, zero dependencies.** The artifact is a Cosmopolitan
   APE: one file runs on Linux/macOS/Windows/BSD, x86_64 and aarch64.
   In the full x86_64 artifact the CUDA backend (`ggml-cuda.so`) is
   embedded and self-extracts to `~/.llamafile/v/<ver>/` on first GPU
   run; the `-win` variant ships without it (see §1.1). TCQ codebooks
   and quantization tables are compiled in. No installer, no downloads.

What upstream gives you is unchanged: the server API
(`/completion`, `/v1/chat/completions`, `/props`, `/slots`, …), GGUF
loading, sampling, chat templates. opencoti adds serving-efficiency
machinery on top, aimed at **multi-session agentic serving on a fixed
VRAM budget**: more concurrent sessions per card, longer usable
context, faster decode.

```bash
chmod +x opencoti-llamafile-0.10.3-c3-x86_64.llamafile
sh ./opencoti-llamafile-0.10.3-c3-x86_64.llamafile --server --port 8080 \
    -m model.gguf -ngl 99 --flash-attn on
# --version → opencoti-0.10.3-c3 ; without --server you get the chat CLI
```

> **Note (Linux):** launch via `sh ./file.llamafile` if your kernel
> lacks binfmt_misc APE registration.

### 1.1 Artifact variants — which file to download

| Artifact | Size | Runs on | GPU story |
|---|---:|---|---|
| `…-x86_64.llamafile` | ~4.8 GB | Linux/macOS/BSD, x86_64 + aarch64 | CUDA DSO **embedded** (x86_64-linux, sm_75/80/86/89/90/120f); self-extracts on first `-ngl` run |
| `…-win-x86_64.llamafile.exe` | ~55 MB | **Windows** + Linux/macOS/BSD, x86_64 + aarch64 | none embedded — CPU works everywhere out of the box; GPU via side-load (below) |
| `…-aarch64.llamafile` *(planned)* | — | same platforms | CUDA sbsa DSO embedded (sm_110f Jetson Thor, sm_121a DGX Spark GB10) |

The host binary inside every variant is byte-for-byte the same APE
(same patches, same features); they differ only in the embedded GPU
backend. The `-win` variant exists because **Windows refuses to run
executables larger than 4 GB**, so the full artifact — 4.8 GB with
the CUDA DSO inside — cannot ship for Windows; the bare APE renamed
to `.exe` (Windows needs the extension) can.

**GPU with the `-win`/bare variant:**

- **Windows:** install the NVIDIA CUDA Toolkit + MSVC and run with
  `-ngl 99`; llamafile compiles a native `ggml-cuda.dll` on the fly
  into `%USERPROFILE%\.llamafile\` on first use (upstream llamafile
  mechanism, unchanged).
- **Linux x86_64:** download the published standalone DSO
  (`dso/<ver>-<tag>/ggml-cuda-x86_64.so` in the HF repo — same bytes
  as the one embedded in the full artifact, sha in
  `releases/<tag>/SHA256SUMS`) and place it where the engine looks
  before attempting extraction:

  ```bash
  mkdir -p ~/.llamafile/v/0.10.3/
  cp ggml-cuda-x86_64.so ~/.llamafile/v/0.10.3/ggml-cuda.so
  ```

- **No CUDA available:** nothing to do — it falls back to CPU
  inference automatically.

---

## 2. Feature map — what exists and how it's gated

| Feature | Default | Turn on with | Class |
|---|---|---|---|
| Session-keyed KV reuse | off (per request) | `session_id` JSON field | latency |
| ReST-KV retention eviction | **off** | `--rest-kv-eviction` | quality-under-overflow |
| KV quantization (scalar) | f16 | `-ctk` / `-ctv` | capacity |
| TurboQuant / TCQ KV tiers | off | `-ctk`/`-ctv turbo*` | capacity |
| Auto KV-tier policy | **off** | `OPENCOTI_KV_AUTO_TIER=1` | capacity (policy) |
| SharedKVPool | off (per request) | `shared_pool_slot` JSON field | multi-agent capacity |
| Rolling-KV window / spill | **auto** (engages only under pressure) | `--vram-target`, `--kv-residency-mode` | capacity |
| Mixed-KV spilled tail | off | `-ctkt` / `-ctvt` | capacity |
| DCA long-context | **off** | `--dca on` | context extension |
| Sparse attention (block-selector) | **off** | `--sparse-attn on` | long-ctx decode speed |
| Sparse-V | auto on iSWA+quant-V, else off | `TURBO_SPARSE_V_TAU` | decode speed |
| MTP speculative decode | **off** | `--spec-type` + drafter | decode speed |
| RYS layer duplication | **off** | `--repeat-layers` | quality |
| RYS probe | off | `--rys-probe` | tooling |
| Lazy slot context | off | `--slot-initial-ctx`, `--slot-shrink-idle-ms` | embedder memory |
| Introspection API | **always on** | `GET /props`, `GET /slots` | observability |

Every boot flag also has an env twin
(`OPENCOTI_LLAMAFILE_<SNAKE_CASE>` for adapter-typed fields,
`LLAMA_ARG_*` for llama.cpp-registered ones).

---

## 3. KV capacity stack

These four features share one goal — **fit more context / more
sessions in fixed VRAM** — and are designed to stack. Recommended
order of adoption: scalar quant → auto-tier → rolling-KV →
turbo tiers → SharedKVPool.

### 3.1 KV quantization: scalar types + TurboQuant/TCQ tiers (PolyKV M6)

The KV cache type is set per-tensor-half: `-ctk <type>` (keys) and
`-ctv <type>` (values), independently — **asymmetric pairs are
first-class** (e.g. `-ctk q8_0 -ctv q4_0`).

Supported types: `f16`, `bf16`, `q8_0`, `q6_0`, `q5_1`, `q5_0`,
`q4_0` (scalar), `turbo2`, `turbo3`, `turbo4`, `turbo8`
(TurboQuant, MSE-optimal with Walsh-Hadamard rotation + InnerQ),
`turbo2_tcq`, `turbo3_tcq` (trellis-coded, Viterbi-encoded).

**Which to pick (measured):**

- **8-bit / 4-bit: use `q8_0` / `q4_0`.** The native scalar types
  dominate turbo8/turbo4 at equal width — turbo earns nothing there.
- **`-ctk q8_0 -ctv q4_0`** is the workhorse asymmetric pair: keys
  keep 8-bit fidelity (attention logits are K-sensitive), values
  take the compression.
- **3 bits and below is TurboQuant territory:** `turbo3`
  Pareto-beats q4_0 (90 vs 129 MiB KV at equal quality, teacher-forced
  TV 0.0067 vs 0.0094); `turbo2` is the smallest logit-equivalent KV
  that exists (~2 bit) — the 256k-context play. TCQ variants trade
  encode cost for a further fidelity step at the same width.
- All shipped tiers pass logit-equivalence gates; decode runs the
  quantized data **in-register** in the flash-attention kernel (no
  f16 materialization) for turbo2/3/4 and TCQ.

Limitations: turbo8 uses a materialize fallback (not fused);
at Gemma-4's head_dim 512 only turbo2/turbo3 have fused D=512
instances; prefill on very long prompts uses a hybrid path
automatically. Quality validation on Gemma franken-merges must use
retrieval (niah), not perplexity.

### 3.2 Auto KV-tier (`OPENCOTI_KV_AUTO_TIER=1`)

Boot policy: pick the **least-compressing scalar pair that keeps the
whole KV resident** in the VRAM budget; if even that spills, the T\*
model decides between "small f16 spill" and "quantize one tier down"
by predicted tokens/s drop.

Knobs (env): `OPENCOTI_KV_AUTO_TIER=1` (master),
`OPENCOTI_KV_TSTAR_DROP` (target drop, default 20%),
`OPENCOTI_KV_TSTAR_MAX_SPILL_MIB` (default 800),
`OPENCOTI_KV_AUTO_TIER_TAIL=1` (also auto-pick a q4_0 spilled tail).
Explicit `-ctv` disables auto entirely; explicit `-ctk` holds K and
walks only V. Dense full-attention models only (iSWA models keep f16).

Read back what it decided: `GET /props → .opencoti.kv.effective` —
the *configured vs effective* split exists exactly because auto-tier
may override you.

### 3.3 Rolling-KV window (residency / spill)

"KV doesn't have to fit." Each layer keeps a device-resident window
of recent positions; the tail `[0 … window_start)` lives in pinned
host RAM and is streamed through the attention kernel per-tile,
merged exactly via online-softmax (LSE). When everything fits, every
layer is `GPU_RESIDENT` and the path is byte-identical to vanilla —
the feature only *engages* under memory pressure.

Flags: `--vram-target <MiB>` (budget cap; `0` = all free VRAM minus
reserve), `--kv-residency-mode {auto,head,window}` (default `auto`;
leave it), `-ctkt` / `-ctvt` (distinct, more-compressed types for the
spilled tail — "mixed-KV": f16 recent window ⊕ q4_0 tail).

Performance model (RTX 3090, PCIe ~6.5 GB/s): spill decode sits at
the PCIe floor, `t(token) ≈ fixed + tail_bytes / link_bw` — linear,
no cliff. On a fast-link host (RTX 6000, ~50 GB/s) window-mode spill
is genuinely usable; on consumer PCIe it's a last resort — prefer
quantizing (that's what auto-tier automates).

Limitations: while a window is spilled, context-shift and
prompt-cache-reuse are guarded off (requests bounded at `n_ctx`);
the compute-buffer reserve for long contexts is measured
automatically at boot (two-pass reserve — no knob).

### 3.4 SharedKVPool (multi-agent shared prefix)

N agents attending **one physical copy** of a common prefix (system
prompt + tool defs). Per-request JSON, no CLI flag:

```jsonc
{ "shared_pool_slot": 0, "shared_prefix_n_tokens": 4096, … }
```

Server must run `--kv-unified --no-clear-idle` (the latter is
mandatory — idle-clear would evict the pooled prefix). Measured:
6.9× KV-cell reduction at N=8 sessions; ~306 agents vs ~9 naive on a
fixed buffer.

Composes with KV quantization (the pool holds quantized cells) and
with session KV-reuse. The pool is read-only for consumers; each
agent's divergent suffix is private.

---

## 4. Long context

### 4.1 DCA — Dual Chunk Attention (training-free context extension)

Splits attention into intra-chunk / successive / inter-chunk position
regimes and merges them exactly by LSE, so a model trained at
`n_ctx_train` serves multiples of it **without retraining**.

Flags: `--dca on` (default **off**), `--dca-chunk-size N` (default
derives from the model's training context; explicit 8192 is the
validated recipe), `--dca-yarn-factor F` (default 1.0; measured
neutral for retrieval — leave it). Serve beyond the GGUF's declared
context with
`--override-kv <arch>.context_length=int:1048576`.

Validated recipe (Gemma-4-A4B, n_ctx_train 256k):

```bash
--dca on --dca-chunk-size 8192 -fa on --parallel 1 \
--override-kv gemma4.context_length=int:1048576
```

Measured retrieval (RULER-VT, n=50): **256k 0.964 · 512k 0.996 ·
768k 0.984 · 1M 0.916** — a gentle −7 pp at 4× native, no cliff.
Counter-proof on Qwen3-8B (native 41k): plain attention collapses at
128k (PPL 19.2) while DCA holds PPL 7.3.

Works on Gemma-4 (its 5 global layers; SWA layers untouched) and
Qwen2.5/3/3.5 (all layers). Composes with quantized KV (scalar pairs
all pass; q8-DCA decode costs ~2× vs f16-DCA), sparse attention, and
rolling-KV.

Limitations: DCA caches K un-rope'd → **launch-time toggle only** (a
server booted DCA-on can't switch off per request); expect
approximation, not identity, past one chunk. On models that are already *native* long-context
(e.g. Qwen2.5-1M), DCA can only approximate down — don't use it
there.

### 4.2 Sparse attention (Quest block-selector) + sparse-V

Two independent decode-bandwidth levers:

- **Block-selector** (`--sparse-attn on`): per-block min/max key
  bounds give an upper bound on each block's attention mass; decode
  visits only the top-K blocks (+ sinks + recent). Flags:
  `--sparse-attn-block-size` (128), `--sparse-attn-topk` (default 0 =
  visit **all** blocks, i.e. no skipping; pass `auto` for adaptive
  max(64, n_blocks/4), or an explicit block count),
  `--sparse-attn-recent`, `--sparse-attn-sink` (1),
  `--sparse-attn-refresh` (8 — re-select every N decode steps),
  `--sparse-attn-mode` (0). Default **off**.
- **Sparse-V**: skips V-dequant for negligible-weight positions
  inside visited blocks. **Self-configuring**: on iSWA models with
  quantized V it auto-sets τ=0.05; elsewhere it stays off. Manual
  override: `TURBO_SPARSE_V_TAU=<float>`.

When to use: **long context on quantized KV.** The win grows with
context (selectivity 0.91@16k → 0.99@40k and climbing) and lives on
quantized KV: q8_0 ⊕ sparse at 50% coverage measured **1.34× decode
at niah 100**. Both levers stack (1.31× combined measured).

When *not* to use: short contexts or f16 KV on mid-size models — the
decode isn't KV-bandwidth-bound there and the selector overhead can
make it *slower* than dense. τ values don't transfer across models;
retune if you override manually.

---

## 5. Decode speed — MTP speculative decoding

Lossless speculative decode; the emitted text is the target model's
own (verified). Two flavours, chosen by `--spec-type`:

### 5.1 `--spec-type draft-assistant` (external drafter — Gemma-4)

A small `gemma4-assistant` drafter GGUF rides the target's
embeddings:

```bash
--spec-type draft-assistant --mtp-head gemma4-assistant-A4B-Q8_0.gguf \
-ngld 99 --spec-draft-n-max 2
```

`--mtp-head` (alias `-md`) names the drafter; **`-ngld 99` matters**
(a CPU-resident draft head erases the win). Drafters for
A4B/12B/27B/E2B/E4B are published per-size. Setting `mtpHead` in the
TS adapter auto-derives the rest.

### 5.2 `--spec-type draft-mtp` (NextN self-spec — Qwen)

Qwen 3.5/3.6 GGUFs that embed a NextN/MTP head self-speculate — no
second file:

```bash
--spec-type draft-mtp --spec-draft-n-max 3
```

Runs per-slot under `--parallel` (multi-session capable).

**Measured (RTX 3090 + upstream-parity campaign):** A4B assistant
decode beats upstream llama.cpp b9859 at every depth (+6.6/+12.9/+7.6%
at n_max 1/2/3); combined with `turbo3_tcq` KV it reaches **~89 tok/s
vs 52.9 plain (+69%)**. Qwen-35B NextN sits at parity with upstream.
Recommended depth: `--spec-draft-n-max 2–3` (A4B), `3` (Qwen NextN).

Notes/limits: acceptance dips a few pp at depth ≥2 (chained-draft
numerics — expected); with `--parallel > 1`, assistant-MTP requires
`--kv-unified` (auto-forced with a boot warning). Composes with
turbo/TCQ KV tiers (its biggest lever), DCA, and quantized KV. Watch
live acceptance per slot via `GET /slots` (§7).

---

## 6. Quality — RYS layer duplication

`--repeat-layers` re-runs a contiguous block of **middle** layers,
weight-shared: zero extra parameter VRAM, no new GGUF, quant-agnostic.
You pay in KV cache and tokens/s proportional to the extra effective
layers; you buy quality-per-token.

```bash
--repeat-layers 33,34      # +1 layer  (RYS-S)
--repeat-layers 26-34      # +8 layers ([26,34) half-open, RYS-XL)
--repeat-layers 8-12;20-24 # disjoint blocks
```

Rules that matter:

- **Middle layers only.** Duplicating first/last layers reliably
  produces incoherent output on merge-fragile models — this is a
  model property, not an engine bug; the engine prints a boot
  advisory when a plan touches the boundary band.
- Absent flag = identity = byte-identical to stock.
- Composes with the full stack: quantized KV, DCA, rolling-KV
  window/spill, sparse-attn (the residency/DCA/sparse sizing paths
  are effective-plan-aware), and MTP — where the draft context
  deliberately runs the un-duplicated base stack while the target
  keeps RYS (still lossless: the target verifies every drafted
  token). Wired across all text archs (dense, MoE, Gemma-4 iSWA
  dual-cache, Qwen 3.5/3.6 recurrent-hybrid); unsupported archs
  fail loudly at load rather than silently ignoring the plan.

**Finding a good plan:** `--rys-probe` enumerates safe-band blocks,
scores each by ΔPPL + a task-probe battery, and prints two
ready-to-paste templates (most-efficient and max-gain):

```bash
sh ./opencoti-llamafile … --rys-probe -m model.gguf -f corpus.txt \
   --rys-probe-widths auto --rys-probe-topk 10
```

Treat its output as a shortlist to verify with your own eval, not a
verdict.

---

## 7. Instrumentation — monitor & control API

Three planes (full reference: `docs/features/introspection.md`):

### Boot knobs
Everything in §§3–6 is a boot flag: set at launch, echoed back at
runtime. By design, tier/residency/DCA/retention **cannot** change
per request (KV layout would differ).

### Per-request control (JSON body fields)

| Field | Default | Effect |
|---|---|---|
| `session_id` | `""` | Session→slot affinity: the same session returns to the slot holding its KV (prevents cross-session eviction at `--parallel > 1`). Pair with `cache_prompt: true`. |
| `shared_pool_slot` | `-1` | Attach this request to SharedKVPool slot N (read-only prefix share). |
| `shared_prefix_n_tokens` | `0` | Length of the shared prefix. |

### Runtime introspection

**`GET /props` → `"opencoti"` object** — boot-state echo plus the
*effective* KV state read back from the live cache:

```jsonc
"opencoti": {
  "kv": { "cache_type_k": "q8_0", "cache_type_v": "q4_0",
          "auto_tier": false,
          "effective": { "type_k": "q8_0", "type_v": "q4_0",
                         "n_cells": 524288, "n_cells_resident": 524288,
                         "n_layers_spilling": 0, "fully_resident": true,
                         "is_iswa": true } },
  "residency":  { "kv_residency_mode": 0, "vram_target_mib": 0 },
  "dca":        { "enabled": true, "chunk_size": 0, "yarn_factor": 1.0 },
  "sparse_attn":{ "enabled": false, "block_size": 128, "topk": 0 },
  "speculative":{ "types": ["none","draft-assistant"], "n_max": 3 },
  "kv_reuse":   { "n_parallel": 4, "kv_unified": true, "cache_ram_mib": 8192 },
  "rest_kv":    { "eviction": false, "recent": 256, "layer": -1 },
  "repeat_layers": null
}
```

`kv.effective` is the only authoritative record of the auto-tier
decision — `configured != effective` is expected when auto-tier
engaged. `fully_resident` / `n_layers_spilling` tell you whether
rolling-KV is streaming.

**`GET /slots` → per-slot `"opencoti"` object** (requires `--slots`):
lifetime `draft_n_total` / `draft_n_accepted` / `draft_acceptance`
per slot, plus the slot's current `session_id` and pool binding.
Operational tell: **sustained draft_acceptance ≳ 0.95 at turn end
usually means the model is looping/ruminating** (healthy agentic
decode sits ~0.4–0.9) — pollable, no log-scraping.

**Per-completion `timings`**: `cache_n` (prefix-reuse hits),
`draft_n` / `draft_n_accepted` for that response.

**Quick recipes**

```bash
curl -s :8080/props | jq .opencoti                      # what is this server running?
curl -s :8080/props | jq .opencoti.kv.effective         # did auto-tier/spill engage?
curl -s :8080/slots | jq '.[] | {id, acc: .opencoti.draft_acceptance}'
```

For embedders/tools linking the C API:
`llama_memory_opencoti_kv_info()` (in `llama.h`) returns the same
effective-KV struct.

### Still log-only
SharedKVPool share/reject events, retention-eviction discards,
rolling-KV tactic selection detail, and the auto-tier WARN line
currently appear only in the server log.

---

## 8. Composition matrix

| | quant-KV | auto-tier | rolling-KV | SharedKVPool | DCA | sparse-attn | MTP | RYS |
|---|---|---|---|---|---|---|---|---|
| **quant-KV** | — | K-only honors | ✅ (tiles dequant-on-lift) | ✅ | ✅ | ✅ (the win case) | ✅ (turbo+MTP is the top decode combo) | ✅ |
| **auto-tier** | | — | ✅ (it *manages* spill) | ✅ | ✅ (probes in DCA state) | ✅ | ✅ | ✅ (sizing is eff-plan-aware) |
| **rolling-KV** | | | — | ✅ | ✅ | ✅ | ✅ | ✅ (validated: window spill × RYS on hybrid) |
| **SharedKVPool** | | | | — | ✅ | ✅ | ✅ | untested |
| **DCA** | | | | | — | ✅ | ✅ (dual-ctx) | ✅ (eff→src mapped) |
| **sparse-attn** | | | | | | — | ✅ | ✅ |
| **MTP** | | | | | | | — | ✅ (draft runs base stack; target keeps RYS) |

Two known guards worth restating: assistant-MTP + `--parallel > 1`
forces `--kv-unified`; SharedKVPool requires
`--kv-unified --no-clear-idle`.

**Reference "agentic serving" launch** (Gemma-4-A4B on a 24 GB card —
quantized KV + MTP + introspection):

```bash
sh ./opencoti-llamafile-0.10.3-c3-x86_64.llamafile --server --port 8080 \
  -m gemma4-A4B-Q4_K_M.gguf -ngl 99 --flash-attn on \
  -c 262144 --parallel 4 --kv-unified \
  -ctk q8_0 -ctv q4_0 \
  --spec-type draft-assistant --mtp-head gemma4-assistant-A4B-Q8_0.gguf \
  -ngld 99 --spec-draft-n-max 2 \
  --slots
```

---

## 9. Internal / superseded machinery (so you don't chase ghosts)

Present in the patch series but **not** user-facing knobs anymore:

- **HeadInfer head-split** (`--headinfer-gpu-heads-frac`): retired as
  a manual knob; it survives as one tactic inside rolling-KV's auto
  ladder (`auto` is the only value you should pass, and the adapter
  does it for you).
- **NEO GPU/CPU FA pipelining** (`--neo-pipeline`): structurally
  shipped, default off; no measurable win on single-GPU consumer
  hardware. Leave off.
- **Fused-MoE up-gate** (`--fused-moe-up-gate`): niche (+2.4% decode
  on OLMoE-class MoE; Gemma-4 already fuses). Default off.
- **Fused-NextN draft graph** (`OPENCOTI_MTP_FUSED_NEXTN=1`): built and
  shipped (patch 0093) but default off for a measured reason — the fused
  graph isn't shape-invariant, so it rebuilds every cycle and decodes
  *slower* than the default autoregressive draft loop (which, post-0128,
  is at upstream parity or better). Leave off.
- **ScoutAttention, LMCache**: design-only / deferred — the flags don't
  exist.

---

## 10. Verifying an artifact

```bash
# 1. artifact hash matches MANIFEST.json / SHA256SUMS
sha256sum opencoti-llamafile-*.llamafile
# 2. embedded CUDA DSO matches the manifest's backend sha — no execution needed
unzip -p opencoti-llamafile-*.llamafile ggml-cuda.so | sha256sum
# 3. version string
sh ./opencoti-llamafile-*.llamafile --version
```

`MANIFEST.json` also records the git commit and the exact patch list
the artifact was built from; the patch series in `patches/`
reproduces the tree from upstream llamafile 0.10.3.