--- license: apache-2.0 tags: - llamafile - llama.cpp - gguf - inference-engine - cuda - opencoti --- # opencoti-llamafile Self-contained single-file inference engine from the [opencoti](https://github.com/mann1x/opencoti) project — a [Mozilla-Ocho llamafile](https://github.com/Mozilla-Ocho/llamafile) 0.10.3 base carrying the opencoti patch series: advanced KV residency/quantization (PolyKV/TurboQuant/TCQ), rolling-KV window with host-RAM spill, DCA long-context extension, MTP speculative decode, sparse attention, RYS layer duplication, and a runtime introspection/control API. **Start with [USAGE.md](./USAGE.md)** — it explains exactly how this engine diverges from upstream llamafile, every added feature, its flags, defaults, limitations, and which features compose. This repo hosts the packaged release artifacts (they exceed GitHub's 2 GiB release-asset cap) plus the full patch series under [`patches/`](./patches) and the project documentation under [`docs/`](./docs). Release notes and per-release `SHA256SUMS` live under [`releases/`](./releases) and are mirrored on the corresponding GitHub release at [mann1x/opencoti](https://github.com/mann1x/opencoti/releases). ## Supported / target model families This engine is **not model-agnostic in what it optimizes**. Upstream llama.cpp GGUF support is unchanged (any GGUF that loads upstream loads here), but the opencoti feature set is built, tuned, and validated on two families: | Family | Role | Models | What's tuned for them | |---|---|---|---| | **Gemma-4** | **PRIMARY target** | 26B-A4B-128e MoE ("A4B"), dense 12B / 31B, elastic E-series (E2B/E4B) | head_dim 256/512 kernels, iSWA dual-cache (rolling-KV, SharedKVPool, DCA wiring), MTP via [gemma4-assistant drafters](https://huggingface.co/ManniX-ITA), fused MoE | | **Qwen** | SECONDARY / verification | Qwen3.5 / 3.6 dense + MoE + hybrid (gated-delta-net), Qwen2.5-14B-1M | head_dim 128 kernels, NextN self-spec MTP (fused multi-step draft), DCA long-context on 1M-class models | Other architectures run with upstream behavior; opencoti features either fall back safely or are unvalidated on them — see the model-families section at the top of [USAGE.md](./USAGE.md) before relying on an opencoti feature elsewhere. ## Artifacts | Artifact | Platform | GPU | |---|---|---| | `opencoti-llamafile---x86_64.llamafile` | x86_64 + aarch64 (fat APE; Linux/macOS/BSD) | CUDA 13.3 **embedded**, x86_64-linux (sm_75/80/86/89/90/120f) | | `opencoti-llamafile---win-x86_64.llamafile.exe` | same fat APE, incl. **Windows** | none embedded — CPU out of the box, GPU via side-load (below) | | `opencoti-llamafile---aarch64.llamafile` | same fat APE | CUDA 13.3 **embedded**, sbsa/aarch64-linux (sm_110f DGX Spark GB10 / Jetson Thor, sm_121a) | | `dso/-/ggml-cuda-x86_64.so` | side-load DSO for the `-win`/bare APE on x86_64-linux | CUDA 13.3, x86_64-linux (sm_75/80/86/89/90/120f) | | `dso/-/ggml-cuda-sbsa-aarch64.so` | side-load DSO for the `-win`/bare APE on aarch64-linux | CUDA 13.3, sbsa/aarch64-linux (sm_110f, sm_121a) | The APE host binary is identical across variants and runs natively on both x86_64 and aarch64. The variants differ ONLY in the embedded `ggml-cuda.so`: the full x86_64 artifact carries the x86_64-linux DSO (4.8 GB); the `-aarch64` artifact carries the sbsa/aarch64-linux DSO (1.8 GB — GPU out of the box on DGX Spark-class hosts); the `-win` variant carries none, because **Windows refuses to run executables larger than 4 GB** — it is the same APE renamed `.exe` (required on Windows) at ~55 MB. With no matching DSO available, inference falls back to CPU. **Release layout:** only the **latest** cut's artifacts live at the repo root (and its DSOs under `dso/-/`). Older cuts are moved whole — artifacts, MANIFESTs, DSOs, notes, SHA256SUMS — to [`releases/archived/`](releases/archived/), one folder per release. Release notes for the current cut are under [`releases/`](releases/). ## Usage ```bash chmod +x opencoti-llamafile-*.llamafile # server mode sh ./opencoti-llamafile-*.llamafile --server --port 8080 \ -m .gguf -ngl 99 --flash-attn on # without --server it starts the interactive chat CLI ``` On the first GPU run the embedded CUDA DSO self-extracts to `~/.llamafile/v//`. `--version` prints the opencoti version string; `GET /props` exposes an `opencoti` introspection section at runtime (see USAGE.md §introspection). ### `-win` variant / GPU side-load - **Windows:** run `opencoti-llamafile---win-x86_64.llamafile.exe` directly (CPU inference works out of the box). For GPU, install the NVIDIA CUDA Toolkit + MSVC; on first `-ngl` run llamafile compiles a native `ggml-cuda.dll` on the fly into `%USERPROFILE%\.llamafile\`. - **Linux x86_64 (bare APE + GPU without the 4.8 GB download’s re-download):** drop the published side-load DSO where the engine looks before extracting an embedded one: ```bash mkdir -p ~/.llamafile/v// # e.g. 0.10.3 cp ggml-cuda-x86_64.so ~/.llamafile/v//ggml-cuda.so ``` Verify against `releases//SHA256SUMS` first. The `-win` file is still a normal APE — `sh ./…​.exe` runs fine on Linux/macOS/BSD. ## Verification Each artifact ships with a `MANIFEST.json` recording the artifact sha256, versionString, gitCommit, the full patch list, and the sha256 of every embedded backend DSO; `releases//SHA256SUMS` is the committed contract (CI verifies this repo's LFS blobs against it on every release tag). Verify the embedded DSO without running: ```bash unzip -p opencoti-llamafile-*.llamafile ggml-cuda.so | sha256sum ``` ## License Apache-2.0 (llamafile) + MIT (llama.cpp and bundled projects). opencoti patches and packaging are MIT. ## PolyKV multi-session stress benchmark (package_courier) An agentic multi-session benchmark: LangGraph courier agents (tool-calling, 10-step delivery chains, 100 packages) run against ONE engine at `--parallel 32`, ctx 163840, q8_0 KV, with a tps-floor scheduler that admits new sessions only while every live session keeps ≥ floor tok/s (PolyKV `/polykv/tps` SSE telemetry drives admission). Each worker episode is a compiled `langgraph.StateGraph` (`capacity_gate → agent ⇄ tools`, with the capacity-gate node reading the engine's `/polykv/capacity`), so the harness doubles as the **reference LangGraph integration** for this engine. The full harness is published in this repo at [`bench/package_courier/`](bench/package_courier/) (work server + LangGraph agent app + launcher + HTML report generator — see its [README](bench/package_courier/README.md) and the usage guide at [`docs/features/package_courier_harness.md`](docs/features/package_courier_harness.md)); reports under [`bench/reports/`](bench/reports/) (per-run HTML + raw stats). The **`opencoti_langgraph` Python package** the harness builds on — `OpencotiChatModel` (LangChain `BaseChatModel` with PolyKV pool/session attach), `PolykvClient`/`AsyncPolykvClient`, pool tools, and the capacity-gate / compaction LangGraph nodes — is published at [`integrations/langgraph/`](integrations/langgraph/) (`pip install ./integrations/langgraph`, needs `langchain-core>=1.4.9` + `langgraph>=1.2.9`; see its [README](integrations/langgraph/README.md)). | model (MTP) | floor | score | wall | delivery mean | fan-out | |---|---|---|---|---|---| | Qwen3.6-27B-Omnimerge-v4 Q6_K (NextN self-spec) | 15 | 99/100 | 2171 s | 146 s | ~7 | | ″ | 5 | 99/100 | 2509 s | 467 s | →22 | | ″ | 1 | 98/100 | 2803 s | 818 s | 32 (max) | | Gemma-4 26B-A4B-it Q4_K_M (assistant-MTP) | 15 | 100/100 | 864 s | 51 s | med 6 | | ″ | 5 | 99/100 | 1090 s | 145 s | med 14 | | ″ | 1 | 95/100 | 1340 s | 346 s | 32 (max) | | ″ (cap 64) | 25 | 100/100 | 773 s | 31 s | ~25 | | ″ (cap 64) | 20 | 100/100 | 823 s | 38 s | ~24 | | Gemma-4 26B-A4B-it **F16** (assistant-MTP, cap 64) | 25 | 100/100 | 1181 s | 35 s | ~32 | | ″ | 20 | 100/100 | 1268 s | 44 s | ~31 | | ″ | 15 | 100/100 | 1369 s | 59 s | ~30 | | ″ | 5 | 100/100 | 1712 s | 174 s | ~34 | | ″ | 1 | 100/100 | 2542 s | 1282 s | 68 (cap band) | | Qwen3.6-35B-A3B UD-Q6_K (NextN self-spec, cap 64) | 25 | 100/100 | 912 s | 55 s | ~21 | | ″ | 20 | 100/100 | 890 s | 64 s | ~28 | | ″ | 15 | 100/100 | 914 s | 84 s | ~24 | | ″ | 5 | 100/100 | 1041 s | 225 s | ~37 | | ″ | 1 | 100/100 | 1574 s | 786 s | 64 (cap) | Aggregate engine throughput stays pinned (Qwen ~75-80 tok/s regardless of concurrency), so equilibrium workers ≈ aggregate/floor. The A4B + assistant-MTP pair is ~2.5× faster end-to-end at every rung. All losses across all runs are agent-side (turn-budget wander on hard packages) — zero tool errors, zero LLM errors, engines clean throughout. The `ab-pooled` vs `ab-naive` pair isolates the SharedKVPool win at identical workload. The A4B F16 ladder answers "what does weight precision cost under agentic load": same engine, same workload, F16 weights are ~1.5× slower end-to-end than Q4_K_M at every floor (bandwidth-bound MoE). At comfortable floors quality is saturated at Q4 (both 100/100), but at max fan-out precision shows: F16 floor 1 finished **zero-loss at 68 workers grinding at ~1.1 tok/s median**, while Q4_K_M at floor 1 loses 3-5 first-wave packages to turn-budget wander (95/100 original; a controlled re-run on the current StateGraph agent, same seed/cap, scored 97/100 with the identical loss signature — confirming the loss mode is model-precision-driven under cold-start contention, not a harness artifact). The 35B-A3B ladder (worker cap raised to 64, floors extended to 20/25) is the cleanest full ladder: **100/100 at every floor including max fan-out**, with the scheduler holding median per-turn tps on the floor (5→5.6, 15→17, 20→23, 25→28) as the fleet shrinks 64→21. It also locates the fan-out knee: wall time is flat (~890-1040 s) from floor 5 up, but floor 1 costs 1574 s — 64 workers oversubscribing 32 slots just queues (per-session tps 2.8, deliveries waiting 786 s mean). Sweet spot for this engine: floor 20-25 — same wall as floor 15 with the best per-package latency at ~21-28 workers. --- Vodafone *Computing resources kindly provided by Vodafone*