docs/features/local_llamafile.md

F2 — Local llamafile Model Serving

Parent: ../MASTER_PLAN.md Status: planning Owner: TBD

Problem

The GPU realization of Tier 0 in the tiered inference engine (F1) wants a local backend that works out of the box — no separate Ollama install required, no separate model server. We want a single binary opencoti can launch on demand.

Beyond out-of-box, the GPU path is where opencoti gets to expose "advanced features" the no-GPU path (Ollama Cloud) can't: KV-cache reuse across agentic turns, big-context throughput, and direct mount of an existing Ollama blob without duplicating the download. The mainline llamafile / llama.cpp do parts of this — opencoti wants a patched build that does all of it.

Goals

• G1. Ship a patched llamafile binary inside opencoti release artifacts.
• G2. Provide a small TypeScript launcher that starts / stops the llamafile, exposes an OpenAI-compatible endpoint to the tier engine, and pins it to loopback by default.
• G3. Maintain patches against upstream llamafile / llama.cpp in vendors/patches/llamafile/, applied at build time.
• G4. Patches focus on three areas:
  • Big-context throughput — keep tokens/sec usable at the context sizes agents actually use (64K+, up to the model's 256K cap).
  • KV-cache reuse across agentic turns — when the new turn shares a prefix with the previous turn (system prompt + tool log), skip reprocessing.
  • Ollama-blob mount — load a GGUF directly from ~/.ollama/models/blobs/sha256-… without copying. Single download, reusable by both Ollama and the patched llamafile.

Non-goals

• Replacing Ollama for users who prefer it as Tier 0. The no-GPU path of F1's Tier 0 explicitly targets Ollama Cloud as an OpenAI-compatible provider — that doesn't go through this package.
• Bundling models. Models are downloaded on first use into the opencoti data dir (or imported from an existing Ollama install); we don't ship weights inside the binary.

Default model

Primary GPU target: ManniX-ITA/gemma-4-A4B-98e-v5-coder-it.

• Architecture: Gemma 4 (MoE), 98-expert prune of Gemma 4 26B-A4B (30 experts dropped per layer, protect_top=16, no shared FFN scaling).
• Total ~20.8 B params, ~4 B active per token.
• Context: 256 K max (recipe defaults to a smaller window for speed but the model supports the full 256 K).
• Pitch: HumanEval 98.17 %, HumanEval+ 92.68 %, LCB-medium 85.45 %. Top of the 14–22 B coder field at this writing.

Quant table (the GGUFs we surface as choices during first-run setup, see F1 M5.7):

Quant	Size	HE+	When to pick
IQ2_S	7.83 GB	85.37%	tight VRAM, 8 GB cards
IQ3_M	9.82 GB	91.46%	default for ≥10 GB VRAM
CD-IQ4_K_M	10.29 GB	92.07%	per-layer "Canary W" variant
IQ4_XS	11.01 GB	93.29%	12 GB cards
Q4_K_S	12.21 GB	93.29%	16 GB cards, good headroom
CD-Q4_K_M_L	13.00 GB	93.29%	per-layer mid-quant variant
Q5_K_L	15.25 GB	93.29%	24 GB cards, room for big context
Q8_0	21.16 GB	93.90%	24 GB cards, maximum quality
F16	39.8 GB	—	reference only

The CD-* variants are "ContribDynamic" per-layer quants from the model card. Both plain and CD- families are presented; the setup flow recommends one based on detected VRAM, the user can override.

Secondary target (deferred): smaller Gemma 4 variants (E2B, E4B, or a non-pruned 26B-A4B). Out of scope until the primary target is working end-to-end. Stock Gemma 3 / Gemma 3 1B are not on the roadmap.

Design sketch

Layout

vendors/
├── sources/
│   └── llamafile/            # upstream llamafile tree (vendored, unmodified)
└── patches/
    └── llamafile/
        ├── 0001-kv-reuse.patch
        ├── 0002-big-context-throughput.patch
        └── README.md         # what each patch does, against which upstream
packages/
└── opencoti-llamafile/       # launcher + adapter for the tier engine
    ├── src/
    │   ├── launch.ts
    │   ├── adapter.ts        # OpenAI-compatible HTTP client wired to tier engine
    │   └── kv-cache.ts       # session-aware cache key derivation
    └── package.json

Pattern: same as Ollama's vendoring of llama.cpp.

Build-time patch application

A small build script under vendors/llamafile/build.{sh,ts} does:

1. git -C vendors/sources/llamafile clean -fdx && git checkout <pinned-sha>
2. Apply every patch under vendors/patches/llamafile/*.patch in lexical order. Bail on conflict.
3. Build (mode depending on target — release CI builds binaries, local dev can use system llamafile if present).
4. Emit the resulting binary to a known location the launcher checks.

The pinned upstream SHA is recorded in vendors/sources/llamafile/UPSTREAM.txt. Updating the SHA is a normal PR — patches that no longer apply must be rebased in the same PR.

Launcher

packages/opencoti-llamafile/src/launch.ts is a thin process manager: starts the binary, waits for the OpenAI-compatible endpoint to respond healthy, exposes port + baseURL. Tears down on opencoti shutdown.

Adapter

packages/opencoti-llamafile/src/adapter.ts implements whatever the tier engine consumes (provider interface — to be defined in F1 M1). It is purely an HTTP client to the local llamafile + cache-key plumbing.

KV-cache reuse — sketch

The agentic loop has a strong invariant: turn N+1 reuses turn N's context prefix verbatim, then appends the model output + new tool result. A KV-reuse patch:

• Adds a session id to the request schema.
• On the server side, persists the KV cache state keyed by (session_id, hash(prefix)).
• On the next request with matching (session_id, hash(prefix)), skips re-prefilling the prefix tokens.

This is conceptually like prompt caching in cloud providers — applied to a local llamafile.

Ollama-blob mount — sketch

If the user already has Ollama installed and has pulled the target model, opencoti should not re-download the GGUF. Ollama stores blobs content-addressed at ~/.ollama/models/blobs/sha256-<hash>, with a manifest at ~/.ollama/models/manifests/registry.ollama.ai/library/<model>/<tag> mapping the model+tag to a digest list (the GGUF blob is one of the layers).

The detection helper (F1 M5.7) parses the manifest and returns the blob path. The patched llamafile accepts that path directly via -m. Single download, no duplication — the same physical file is used by both Ollama and llamafile.

The patch in llamafile itself is small: it already accepts -m <path>, but blob filenames are hash-named (no .gguf extension), so any code path that gates on file-extension needs adjusting. That's the patch.

Milestones

M1 — Llamafile vendor present, launcher boots it (done — 2026-05-21)

• Vendor llamafile under vendors/sources/llamafile/ (git submodule pinned to 0.10.1 / 6490e16).
• vendors/patches/llamafile/ exists with README.md documenting the patch protocol (numeric prefixes, lexical apply order, upstreaming policy). No patches yet.
• packages/opencoti-llamafile/ (@opencoti/llamafile) ships config.ts (binary resolution: env → vendored build path → PATH), launch.ts (Bun.spawn + waitForReady polling /v1/models), adapter.ts (M2 stub), and script/smoke.ts. Typecheck is clean.
• Smoke script exits 2 with a clear diagnostic when no binary is resolvable, and 0 when one is. Live --server launch + /v1/models ping is gated on OPENCOTI_LLAMAFILE_MODEL being set.

M2 — Adapter wired to tier engine (done — 2026-05-21)

• packages/opencoti-llamafile/src/tier-provider.ts exports tierProvider() returning a TierProvider (the interface defined by F1 M3 in @opencoti/tiers/provider).
• Implementation uses @ai-sdk/openai-compatible to build a LanguageModel against the running llamafile's /v1 endpoint; the previous M1 stub adapter.ts is preserved alongside (it's used by the smoke script).
• Singleton llamafile process per host (runningPromise module cache); cleanup is process-exit (good enough for M3, refined later).
• @opencoti/llamafile/package.json declares deps on @ai-sdk/openai-compatible and @opencoti/tiers (for the interface).
• F1 M3 (Tier 0 = llamafile end-to-end) is achieved jointly with this milestone.

M3 — Build pipeline applies patches (done — 2026-05-22)

Pure-helpers + CLI orchestrator split (same testability pattern as M5.7's setup-flow):

• packages/opencoti-llamafile/src/build-pipeline.ts (new) — pure helpers: discoverPatches, validatePatchName, summarizePatchSet, readPinnedSha. Re-exported from the package and from a new ./build-pipeline subpath.
• packages/opencoti-llamafile/script/build-pipeline.ts (new) — CLI orchestrator with subcommands check, reset, apply, build, plus help. Shells to git + make via Bun.spawn. Detects uninitialized submodule and emits a clear "run git submodule update --init" message.
• vendors/pin/llamafile.txt (new) — pinned SHA + tag, machine-readable. Lives OUTSIDE the submodule so git clean -fdx doesn't wipe it. The user-facing pin table in vendors/README.md is kept in sync manually.
• Root package.json scripts: build:llamafile, build:llamafile:check, build:llamafile:apply, build:llamafile:reset.

Patch protocol enforced:

• NNNN-<kebab>.patch with 4-digit prefix and lowercase kebab title.
• Duplicate prefixes are flagged (both names appear in the invalid list, neither in patches).
• Empty patch set produces "no patches" — the build still runs to completion.
• check warns when the submodule HEAD has drifted from the pin but doesn't fail (a developer may be intentionally bumping).

Test coverage: 21 new unit tests in packages/opencoti-llamafile/test/build-pipeline.test.ts covering all helpers + the fixture-based discover path. Fixtures live under test/fixtures/patches/ (0001-foo.patch, 0002-bar-baz.patch, README.md to exercise the non-.patch-ignore path). The CLI orchestrator itself is NOT unit-tested (it shells out to git/make; integration testing belongs to F2 M6).

Verification: bun run build:llamafile:check against the current empty patch set:

[build-pipeline] Pinned upstream: 6490e16
[build-pipeline] Vendor HEAD:     6490e16f7a8fb7d8a0a0734b41490c3f8db78763
[build-pipeline] no patches
[build-pipeline] check OK

Zero new surgical hooks. Hook footprint stays at 4 markers.

M4 — First patch: KV-cache reuse v1 (absorbed into F5 M0 — shipped 2026-05-24)

This milestone was absorbed into the F5 advanced-KV series as F5 M0 (homologous to the rest of that patch series) and has shipped. It landed as vendors/patches/llamafile/0006-kv-reuse-prefix.patch (not 0001-, since F4 M3's lazy-slot work consumed 0001–0005): an optional session_id request field + session→slot affinity, with the adapter threading the session id and cache_prompt through a fetch wrapper. Bench perf/llamafile/turn-2-latency.bench.ts shows turn-2 prefill collapsing from 2061→20 tokens. See docs/features/advanced_kv.md "M0" for the design, the upstream-already-reuses finding, and the deferred prefix-hash/disk extension.

M5 — Big-context throughput patches

• Targeted micro-patches that improve tokens/sec at 64K+ contexts (up to the 256K model cap).
• Each patch documented in vendors/patches/llamafile/README.md.
• Each patch with a benchmark in perf/ (we can reuse the existing perf/ dir from opencode).

M6 — Release artifact bundling

• CI builds llamafile-with-patches for the platforms opencoti ships on. Binary is included in the opencoti release tarball under a known path the launcher will find.

M7 — Ollama-blob mount patch

• Patch lands in vendors/patches/llamafile/0002-ollama-blob-mount.patch.
• The patched llamafile accepts hash-named (extension-less) GGUF paths from ~/.ollama/models/blobs/sha256-… directly.
• Wired into F1 M5.7's first-run setup flow as the "Import from local Ollama" option.
• Smoke test: install Ollama, ollama pull gemma3:1b (or any test model), point the patched llamafile at the blob, assert health.

Open questions

• Llamafile vs. raw llama.cpp. Llamafile gives us the single-binary-cross-platform property. If KV-reuse patching is easier against llama.cpp upstream, we may switch vendors. Still open after M1 — re-evaluate when the first patch (M4) is in flight.
• Default model. Qwen2.5-Coder-7B vs Qwen2.5-Coder-14B. Resolved: ManniX-ITA/gemma-4-A4B-98e-v5-coder-it (see Default model section above). Smaller Gemma 4 variants are explicitly deferred secondary targets.
• Sandboxing. Launching a long-running binary needs robust shutdown. Use a process group + signal forwarding so an opencoti crash doesn't leave a llamafile orphan.
• head_dim=512 constraint. Gemma 4 has head_dim=512, which means FlashAttention 2 is NOT supported (per the model card). Patches need to leave the attn_implementation="eager" path intact and not enable FA2 indiscriminately for big-context work.

Risks

• Patches drift when upstream moves. Mitigation: pinned SHA + the rebase-on-update rule, and per-patch tests.
• Binary size. Llamafile binaries are large. Mitigation: ship the binary as a separate artifact downloaded on first run, not inside the main opencoti package, if it gets prohibitive.