# F2 — Local llamafile Model Serving

> Parent: [../MASTER_PLAN.md](../MASTER_PLAN.md)
> Status: **planning**
> Owner: TBD

## Problem

The GPU realization of Tier 0 in the tiered inference engine
([F1](tiered_inference.md)) wants a local backend that works **out of
the box** — no separate Ollama install required, no separate model
server. We want a single binary opencoti can launch on demand.

Beyond out-of-box, the GPU path is where opencoti gets to expose
"advanced features" the no-GPU path (Ollama Cloud) can't: KV-cache
reuse across agentic turns, big-context throughput, and direct mount
of an existing Ollama blob without duplicating the download. The
mainline llamafile / llama.cpp do parts of this — opencoti wants a
patched build that does all of it.

## Goals

- **G1.** Ship a patched llamafile binary inside opencoti release
  artifacts.
- **G2.** Provide a small TypeScript launcher that starts / stops the
  llamafile, exposes an OpenAI-compatible endpoint to the tier
  engine, and pins it to loopback by default.
- **G3.** Maintain patches against upstream llamafile / llama.cpp in
  `vendors/patches/llamafile/`, applied at build time.
- **G4.** Patches focus on three areas:
  - **Big-context throughput** — keep tokens/sec usable at the
    context sizes agents actually use (64K+, up to the model's 256K cap).
  - **KV-cache reuse across agentic turns** — when the new turn
    shares a prefix with the previous turn (system prompt + tool
    log), skip reprocessing.
  - **Ollama-blob mount** — load a GGUF directly from
    `~/.ollama/models/blobs/sha256-…` without copying. Single
    download, reusable by both Ollama and the patched llamafile.

## Non-goals

- Replacing Ollama for users who prefer it as Tier 0. The no-GPU
  path of F1's Tier 0 explicitly targets Ollama Cloud as an
  OpenAI-compatible provider — that doesn't go through this package.
- Bundling models. Models are downloaded on first use into the
  opencoti data dir (or imported from an existing Ollama install);
  we don't ship weights inside the binary.

## Default model

**Primary GPU target:**
[`ManniX-ITA/gemma-4-A4B-98e-v5-coder-it`](https://huggingface.co/ManniX-ITA/gemma-4-A4B-98e-v5-coder-it-GGUF).

- Architecture: Gemma 4 (MoE), 98-expert prune of Gemma 4 26B-A4B
  (30 experts dropped per layer, `protect_top=16`, no shared FFN
  scaling).
- Total ~20.8 B params, ~4 B active per token.
- Context: **256 K max** (recipe defaults to a smaller window for
  speed but the model supports the full 256 K).
- Pitch: HumanEval 98.17 %, HumanEval+ 92.68 %, LCB-medium 85.45 %.
  Top of the 14–22 B coder field at this writing.

**Quant table** (the GGUFs we surface as choices during first-run
setup, see F1 M5.7):

| Quant       | Size     | HE+    | When to pick                          |
|-------------|----------|--------|----------------------------------------|
| IQ2_S       | 7.83 GB  | 85.37% | tight VRAM, 8 GB cards                 |
| IQ3_M       | 9.82 GB  | 91.46% | **default for ≥10 GB VRAM**            |
| CD-IQ4_K_M  | 10.29 GB | 92.07% | per-layer "Canary W" variant           |
| IQ4_XS      | 11.01 GB | 93.29% | 12 GB cards                            |
| Q4_K_S      | 12.21 GB | 93.29% | 16 GB cards, good headroom             |
| CD-Q4_K_M_L | 13.00 GB | 93.29% | per-layer mid-quant variant            |
| Q5_K_L      | 15.25 GB | 93.29% | 24 GB cards, room for big context      |
| Q8_0        | 21.16 GB | 93.90% | 24 GB cards, maximum quality           |
| F16         | 39.8 GB  | —      | reference only                         |

The CD-* variants are "ContribDynamic" per-layer quants from the
model card. Both plain and CD- families are presented; the setup
flow recommends one based on detected VRAM, the user can override.

**Secondary target (deferred):** smaller Gemma 4 variants (E2B, E4B,
or a non-pruned 26B-A4B). Out of scope until the primary target is
working end-to-end. Stock Gemma 3 / Gemma 3 1B are not on the
roadmap.

## Design sketch

### Layout

```
vendors/
├── sources/
│   └── llamafile/            # upstream llamafile tree (vendored, unmodified)
└── patches/
    └── llamafile/
        ├── 0001-kv-reuse.patch
        ├── 0002-big-context-throughput.patch
        └── README.md         # what each patch does, against which upstream
packages/
└── opencoti-llamafile/       # launcher + adapter for the tier engine
    ├── src/
    │   ├── launch.ts
    │   ├── adapter.ts        # OpenAI-compatible HTTP client wired to tier engine
    │   └── kv-cache.ts       # session-aware cache key derivation
    └── package.json
```

Pattern: same as Ollama's vendoring of `llama.cpp`.

### Build-time patch application

A small build script under `vendors/llamafile/build.{sh,ts}` does:

1. `git -C vendors/sources/llamafile clean -fdx && git checkout <pinned-sha>`
2. Apply every patch under `vendors/patches/llamafile/*.patch` in
   lexical order. Bail on conflict.
3. Build (mode depending on target — release CI builds binaries,
   local dev can use system llamafile if present).
4. Emit the resulting binary to a known location the launcher checks.

The pinned upstream SHA is recorded in
`vendors/sources/llamafile/UPSTREAM.txt`. Updating the SHA is a
normal PR — patches that no longer apply must be rebased in the same
PR.

### Launcher

`packages/opencoti-llamafile/src/launch.ts` is a thin process
manager: starts the binary, waits for the OpenAI-compatible endpoint
to respond healthy, exposes `port` + `baseURL`. Tears down on
opencoti shutdown.

### Adapter

`packages/opencoti-llamafile/src/adapter.ts` implements whatever the
tier engine consumes (provider interface — to be defined in F1 M1).
It is purely an HTTP client to the local llamafile + cache-key
plumbing.

### KV-cache reuse — sketch

The agentic loop has a strong invariant: turn N+1 reuses turn N's
context prefix verbatim, then appends the model output + new tool
result. A KV-reuse patch:

- Adds a session id to the request schema.
- On the server side, persists the KV cache state keyed by
  `(session_id, hash(prefix))`.
- On the next request with matching `(session_id, hash(prefix))`,
  skips re-prefilling the prefix tokens.

This is conceptually like prompt caching in cloud providers — applied
to a local llamafile.

### Ollama-blob mount — sketch

If the user already has Ollama installed and has pulled the target
model, opencoti should not re-download the GGUF. Ollama stores blobs
content-addressed at `~/.ollama/models/blobs/sha256-<hash>`, with a
manifest at
`~/.ollama/models/manifests/registry.ollama.ai/library/<model>/<tag>`
mapping the model+tag to a digest list (the GGUF blob is one of the
layers).

The detection helper (F1 M5.7) parses the manifest and returns the
blob path. The patched llamafile accepts that path directly via
`-m`. Single download, no duplication — the same physical file is
used by both Ollama and llamafile.

The patch in llamafile itself is small: it already accepts `-m
<path>`, but blob filenames are hash-named (no `.gguf` extension), so
any code path that gates on file-extension needs adjusting. That's
the patch.

## Milestones

### M1 — Llamafile vendor present, launcher boots it *(done — 2026-05-21)*

- [x] Vendor llamafile under `vendors/sources/llamafile/` (git
  submodule pinned to `0.10.1` / `6490e16`).
- [x] `vendors/patches/llamafile/` exists with `README.md`
  documenting the patch protocol (numeric prefixes, lexical apply
  order, upstreaming policy). No patches yet.
- [x] `packages/opencoti-llamafile/` (`@opencoti/llamafile`) ships
  `config.ts` (binary resolution: env → vendored build path → PATH),
  `launch.ts` (`Bun.spawn` + `waitForReady` polling `/v1/models`),
  `adapter.ts` (M2 stub), and `script/smoke.ts`. Typecheck is clean.
- Smoke script exits 2 with a clear diagnostic when no binary is
  resolvable, and 0 when one is. Live `--server` launch + `/v1/models`
  ping is gated on `OPENCOTI_LLAMAFILE_MODEL` being set.

### M2 — Adapter wired to tier engine *(done — 2026-05-21)*

- [x] `packages/opencoti-llamafile/src/tier-provider.ts` exports
  `tierProvider()` returning a `TierProvider` (the interface
  defined by F1 M3 in `@opencoti/tiers/provider`).
- [x] Implementation uses `@ai-sdk/openai-compatible` to build a
  `LanguageModel` against the running llamafile's `/v1` endpoint;
  the previous M1 stub `adapter.ts` is preserved alongside (it's
  used by the smoke script).
- [x] Singleton llamafile process per host (`runningPromise` module
  cache); cleanup is process-exit (good enough for M3, refined
  later).
- [x] `@opencoti/llamafile/package.json` declares deps on
  `@ai-sdk/openai-compatible` and `@opencoti/tiers` (for the
  interface).
- [x] F1 M3 (Tier 0 = llamafile end-to-end) is achieved jointly
  with this milestone.

### M3 — Build pipeline applies patches *(done — 2026-05-22)*

Pure-helpers + CLI orchestrator split (same testability pattern as
M5.7's setup-flow):

- `packages/opencoti-llamafile/src/build-pipeline.ts` (new) —
  pure helpers: `discoverPatches`, `validatePatchName`,
  `summarizePatchSet`, `readPinnedSha`. Re-exported from the
  package and from a new `./build-pipeline` subpath.
- `packages/opencoti-llamafile/script/build-pipeline.ts` (new) —
  CLI orchestrator with subcommands `check`, `reset`, `apply`,
  `build`, plus `help`. Shells to git + make via `Bun.spawn`.
  Detects uninitialized submodule and emits a clear "run
  `git submodule update --init`" message.
- `vendors/pin/llamafile.txt` (new) — pinned SHA + tag,
  machine-readable. Lives OUTSIDE the submodule so
  `git clean -fdx` doesn't wipe it. The user-facing pin table in
  `vendors/README.md` is kept in sync manually.
- Root `package.json` scripts: `build:llamafile`,
  `build:llamafile:check`, `build:llamafile:apply`,
  `build:llamafile:reset`.

Patch protocol enforced:

- `NNNN-<kebab>.patch` with 4-digit prefix and lowercase kebab title.
- Duplicate prefixes are flagged (both names appear in the
  `invalid` list, neither in `patches`).
- Empty patch set produces "no patches" — the build still runs to
  completion.
- `check` warns when the submodule HEAD has drifted from the pin
  but doesn't fail (a developer may be intentionally bumping).

Test coverage: 21 new unit tests in
`packages/opencoti-llamafile/test/build-pipeline.test.ts` covering
all helpers + the fixture-based discover path. Fixtures live under
`test/fixtures/patches/` (`0001-foo.patch`, `0002-bar-baz.patch`,
README.md to exercise the non-`.patch`-ignore path). The CLI
orchestrator itself is NOT unit-tested (it shells out to git/make;
integration testing belongs to F2 M6).

Verification: `bun run build:llamafile:check` against the current
empty patch set:
```
[build-pipeline] Pinned upstream: 6490e16
[build-pipeline] Vendor HEAD:     6490e16f7a8fb7d8a0a0734b41490c3f8db78763
[build-pipeline] no patches
[build-pipeline] check OK
```

Zero new surgical hooks. Hook footprint stays at 4 markers.

### M4 — First patch: KV-cache reuse v1 *(absorbed into F5 M0 — shipped 2026-05-24)*

This milestone was absorbed into the F5 advanced-KV series as **F5 M0**
(homologous to the rest of that patch series) and has **shipped**. It landed
as `vendors/patches/llamafile/0006-kv-reuse-prefix.patch` (not `0001-`, since
F4 M3's lazy-slot work consumed 0001–0005): an optional `session_id` request
field + session→slot affinity, with the adapter threading the session id and
`cache_prompt` through a `fetch` wrapper. Bench
`perf/llamafile/turn-2-latency.bench.ts` shows turn-2 prefill collapsing from
2061→20 tokens. See
[docs/features/advanced_kv.md](advanced_kv.md) "M0" for the design, the
upstream-already-reuses finding, and the deferred prefix-hash/disk extension.

### M5 — Big-context throughput patches

- Targeted micro-patches that improve tokens/sec at 64K+ contexts
  (up to the 256K model cap).
- Each patch documented in `vendors/patches/llamafile/README.md`.
- Each patch with a benchmark in `perf/` (we can reuse the existing
  `perf/` dir from opencode).

### M6 — Release artifact bundling

- CI builds llamafile-with-patches for the platforms opencoti ships
  on. Binary is included in the opencoti release tarball under a
  known path the launcher will find.

### M7 — Ollama-blob mount patch

- Patch lands in `vendors/patches/llamafile/0002-ollama-blob-mount.patch`.
- The patched llamafile accepts hash-named (extension-less) GGUF
  paths from `~/.ollama/models/blobs/sha256-…` directly.
- Wired into F1 M5.7's first-run setup flow as the "Import from
  local Ollama" option.
- Smoke test: install Ollama, `ollama pull gemma3:1b` (or any test
  model), point the patched llamafile at the blob, assert health.

## Open questions

- **Llamafile vs. raw llama.cpp.** Llamafile gives us the
  single-binary-cross-platform property. If KV-reuse patching is
  easier against llama.cpp upstream, we may switch vendors. Still
  open after M1 — re-evaluate when the first patch (M4) is in flight.
- ~~**Default model.** Qwen2.5-Coder-7B vs Qwen2.5-Coder-14B.~~
  Resolved: `ManniX-ITA/gemma-4-A4B-98e-v5-coder-it` (see *Default
  model* section above). Smaller Gemma 4 variants are explicitly
  deferred secondary targets.
- **Sandboxing.** Launching a long-running binary needs robust
  shutdown. Use a process group + signal forwarding so an opencoti
  crash doesn't leave a llamafile orphan.
- **`head_dim=512` constraint.** Gemma 4 has `head_dim=512`, which
  means FlashAttention 2 is NOT supported (per the model card).
  Patches need to leave the `attn_implementation="eager"` path
  intact and not enable FA2 indiscriminately for big-context work.

## Risks

- Patches drift when upstream moves. Mitigation: pinned SHA + the
  rebase-on-update rule, and per-patch tests.
- Binary size. Llamafile binaries are large. Mitigation: ship the
  binary as a separate artifact downloaded on first run, not inside
  the main opencoti package, if it gets prohibitive.