README.md

---
license: apache-2.0
tags:
  - llamafile
  - llama.cpp
  - gguf
  - inference-engine
  - cuda
  - opencoti
---

# opencoti-llamafile

Self-contained single-file inference engine from the
[opencoti](https://github.com/mann1x/opencoti) project — a
[Mozilla-Ocho llamafile](https://github.com/Mozilla-Ocho/llamafile)
0.10.3 base carrying the opencoti patch series: advanced KV
residency/quantization (PolyKV/TurboQuant/TCQ), rolling-KV window with
host-RAM spill, DCA long-context extension, MTP speculative decode,
sparse attention, RYS layer duplication, and a runtime
introspection/control API.

**Start with [USAGE.md](./USAGE.md)** — it explains exactly how this
engine diverges from upstream llamafile, every added feature, its
flags, defaults, limitations, and which features compose.

This repo hosts the packaged release artifacts (they exceed GitHub's
2 GiB release-asset cap) plus the full patch series under
[`patches/`](./patches) and the project documentation under
[`docs/`](./docs). Release notes and per-release `SHA256SUMS` live
under [`releases/`](./releases) and are mirrored on the corresponding
GitHub release at
[mann1x/opencoti](https://github.com/mann1x/opencoti/releases).

## Supported / target model families

This engine is **not model-agnostic in what it optimizes**. Upstream
llama.cpp GGUF support is unchanged (any GGUF that loads upstream
loads here), but the opencoti feature set is built, tuned, and
validated on two families:

| Family | Role | Models | What's tuned for them |
|---|---|---|---|
| **Gemma-4** | **PRIMARY target** | 26B-A4B-128e MoE ("A4B"), dense 12B / 31B, elastic E-series (E2B/E4B) | head_dim 256/512 kernels, iSWA dual-cache (rolling-KV, SharedKVPool, DCA wiring), MTP via [gemma4-assistant drafters](https://huggingface.co/ManniX-ITA), fused MoE |
| **Qwen** | SECONDARY / verification | Qwen3.5 / 3.6 dense + MoE + hybrid (gated-delta-net), Qwen2.5-14B-1M | head_dim 128 kernels, NextN self-spec MTP (fused multi-step draft), DCA long-context on 1M-class models |

Other architectures run with upstream behavior; opencoti features
either fall back safely or are unvalidated on them — see the
model-families section at the top of [USAGE.md](./USAGE.md) before
relying on an opencoti feature elsewhere.

## Artifacts

| Artifact | Platform | GPU |
|---|---|---|
| `opencoti-llamafile-<ver>-<tag>-x86_64.llamafile` | x86_64 + aarch64 (fat APE; Linux/macOS/BSD) | CUDA 13.3 **embedded**, x86_64-linux (sm_75/80/86/89/90/120f) |
| `opencoti-llamafile-<ver>-<tag>-win-x86_64.llamafile.exe` | same fat APE, incl. **Windows** | none embedded — CPU out of the box, GPU via side-load (below) |
| `opencoti-llamafile-<ver>-<tag>-aarch64.llamafile` | same fat APE | CUDA 13.3 **embedded**, sbsa/aarch64-linux (sm_110f DGX Spark GB10 / Jetson Thor, sm_121a) |
| `dso/<ver>-<tag>/ggml-cuda-x86_64.so` | side-load DSO for the `-win`/bare APE on x86_64-linux | CUDA 13.3, x86_64-linux (sm_75/80/86/89/90/120f) |
| `dso/<ver>-<tag>/ggml-cuda-sbsa-aarch64.so` | side-load DSO for the `-win`/bare APE on aarch64-linux | CUDA 13.3, sbsa/aarch64-linux (sm_110f, sm_121a) |

The APE host binary is identical across variants and runs natively on
both x86_64 and aarch64. The variants differ ONLY in the embedded
`ggml-cuda.so`: the full x86_64 artifact carries the x86_64-linux DSO
(4.8 GB); the `-aarch64` artifact carries the sbsa/aarch64-linux DSO
(1.8 GB — GPU out of the box on DGX Spark-class hosts); the `-win`
variant carries none, because **Windows refuses to run executables
larger than 4 GB** — it is the same APE renamed `.exe` (required on
Windows) at ~55 MB. With no matching DSO available, inference falls
back to CPU.

**Release layout:** only the **latest** cut's artifacts live at the repo
root (and its DSOs under `dso/<ver>-<tag>/`). Older cuts are moved
whole — artifacts, MANIFESTs, DSOs, notes, SHA256SUMS — to
[`releases/archived/`](releases/archived/), one folder per release.
Release notes for the current cut are under
[`releases/`](releases/).

## Usage

```bash
chmod +x opencoti-llamafile-*.llamafile
# server mode
sh ./opencoti-llamafile-*.llamafile --server --port 8080 \
    -m <model>.gguf -ngl 99 --flash-attn on
# without --server it starts the interactive chat CLI
```

On the first GPU run the embedded CUDA DSO self-extracts to
`~/.llamafile/v/<ver>/`. `--version` prints the opencoti version
string; `GET /props` exposes an `opencoti` introspection section at
runtime (see USAGE.md §introspection).

### `-win` variant / GPU side-load

- **Windows:** run `opencoti-llamafile-<ver>-<tag>-win-x86_64.llamafile.exe`
  directly (CPU inference works out of the box). For GPU, install the
  NVIDIA CUDA Toolkit + MSVC; on first `-ngl` run llamafile compiles a
  native `ggml-cuda.dll` on the fly into `%USERPROFILE%\.llamafile\`.
- **Linux x86_64 (bare APE + GPU without the 4.8 GB download’s
  re-download):** drop the published side-load DSO where the engine
  looks before extracting an embedded one:

  ```bash
  mkdir -p ~/.llamafile/v/<ver>/   # e.g. 0.10.3
  cp ggml-cuda-x86_64.so ~/.llamafile/v/<ver>/ggml-cuda.so
  ```

  Verify against `releases/<tag>/SHA256SUMS` first. The `-win` file is
  still a normal APE — `sh ./…​.exe` runs fine on Linux/macOS/BSD.

## Verification

Each artifact ships with a `MANIFEST.json` recording the artifact
sha256, versionString, gitCommit, the full patch list, and the sha256
of every embedded backend DSO; `releases/<tag>/SHA256SUMS` is the
committed contract (CI verifies this repo's LFS blobs against it on
every release tag). Verify the embedded DSO without running:

```bash
unzip -p opencoti-llamafile-*.llamafile ggml-cuda.so | sha256sum
```

## License

Apache-2.0 (llamafile) + MIT (llama.cpp and bundled projects).
opencoti patches and packaging are MIT.


## PolyKV multi-session stress benchmark (package_courier)

An agentic multi-session benchmark: LangGraph courier agents (tool-calling,
10-step delivery chains, 100 packages) run against ONE engine at
`--parallel 32`, ctx 163840, q8_0 KV, with a tps-floor scheduler that admits
new sessions only while every live session keeps ≥ floor tok/s (PolyKV
`/polykv/tps` SSE telemetry drives admission). Each worker episode is a
compiled `langgraph.StateGraph` (`capacity_gate → agent ⇄ tools`, with the
capacity-gate node reading the engine's `/polykv/capacity`), so the harness
doubles as the **reference LangGraph integration** for this engine. The full
harness is published in this repo at
[`bench/package_courier/`](bench/package_courier/)
(work server + LangGraph agent app + launcher + HTML report generator —
see its [README](bench/package_courier/README.md) and the usage guide at
[`docs/features/package_courier_harness.md`](docs/features/package_courier_harness.md));
reports under [`bench/reports/`](bench/reports/) (per-run HTML + raw stats).

The **`opencoti_langgraph` Python package** the harness builds on —
`OpencotiChatModel` (LangChain `BaseChatModel` with PolyKV pool/session
attach), `PolykvClient`/`AsyncPolykvClient`, pool tools, and the
capacity-gate / compaction LangGraph nodes — is published at
[`integrations/langgraph/`](integrations/langgraph/)
(`pip install ./integrations/langgraph`, needs `langchain-core>=1.4.9`
+ `langgraph>=1.2.9`; see its [README](integrations/langgraph/README.md)).

| model (MTP) | floor | score | wall | delivery mean | fan-out |
|---|---|---|---|---|---|
| Qwen3.6-27B-Omnimerge-v4 Q6_K (NextN self-spec) | 15 | 99/100 | 2171 s | 146 s | ~7 |
| ″ | 5 | 99/100 | 2509 s | 467 s | →22 |
| ″ | 1 | 98/100 | 2803 s | 818 s | 32 (max) |
| Gemma-4 26B-A4B-it Q4_K_M (assistant-MTP) | 15 | 100/100 | 864 s | 51 s | med 6 |
| ″ | 5 | 99/100 | 1090 s | 145 s | med 14 |
| ″ | 1 | 95/100 | 1340 s | 346 s | 32 (max) |
| ″ (cap 64) | 25 | 100/100 | 773 s | 31 s | ~25 |
| ″ (cap 64) | 20 | 100/100 | 823 s | 38 s | ~24 |
| Gemma-4 26B-A4B-it **F16** (assistant-MTP, cap 64) | 25 | 100/100 | 1181 s | 35 s | ~32 |
| ″ | 20 | 100/100 | 1268 s | 44 s | ~31 |
| ″ | 15 | 100/100 | 1369 s | 59 s | ~30 |
| ″ | 5 | 100/100 | 1712 s | 174 s | ~34 |
| ″ | 1 | 100/100 | 2542 s | 1282 s | 68 (cap band) |
| Qwen3.6-35B-A3B UD-Q6_K (NextN self-spec, cap 64) | 25 | 100/100 | 912 s | 55 s | ~21 |
| ″ | 20 | 100/100 | 890 s | 64 s | ~28 |
| ″ | 15 | 100/100 | 914 s | 84 s | ~24 |
| ″ | 5 | 100/100 | 1041 s | 225 s | ~37 |
| ″ | 1 | 100/100 | 1574 s | 786 s | 64 (cap) |

Aggregate engine throughput stays pinned (Qwen ~75-80 tok/s regardless of
concurrency), so equilibrium workers ≈ aggregate/floor. The A4B + assistant-MTP
pair is ~2.5× faster end-to-end at every rung. All losses across all runs are
agent-side (turn-budget wander on hard packages) — zero tool errors, zero LLM
errors, engines clean throughout. The `ab-pooled` vs `ab-naive` pair isolates
the SharedKVPool win at identical workload.

The A4B F16 ladder answers "what does weight precision cost under agentic
load": same engine, same workload, F16 weights are ~1.5× slower end-to-end
than Q4_K_M at every floor (bandwidth-bound MoE). At comfortable floors
quality is saturated at Q4 (both 100/100), but at max fan-out precision
shows: F16 floor 1 finished **zero-loss at 68 workers grinding at ~1.1
tok/s median**, while Q4_K_M at floor 1 loses 3-5 first-wave packages to
turn-budget wander (95/100 original; a controlled re-run on the current
StateGraph agent, same seed/cap, scored 97/100 with the identical loss
signature — confirming the loss mode is model-precision-driven under
cold-start contention, not a harness artifact).

The 35B-A3B ladder (worker cap raised to 64, floors extended to 20/25) is the
cleanest full ladder: **100/100 at every floor including max fan-out**, with
the scheduler holding median per-turn tps on the floor (5→5.6, 15→17, 20→23,
25→28) as the fleet shrinks 64→21. It also locates the fan-out knee: wall time
is flat (~890-1040 s) from floor 5 up, but floor 1 costs 1574 s — 64 workers
oversubscribing 32 slots just queues (per-session tps 2.8, deliveries waiting
786 s mean). Sweet spot for this engine: floor 20-25 — same wall as floor 15
with the best per-package latency at ~21-28 workers.

---

<img src="vodafone-logo.png" alt="Vodafone" width="96"/>

*Computing resources kindly provided by Vodafone*