--- license: apache-2.0 tags: - llamafile - llama.cpp - gguf - inference-engine - cuda - opencoti --- # opencoti-llamafile Self-contained single-file inference engine from the [opencoti](https://github.com/mann1x/opencoti) project — a [Mozilla-Ocho llamafile](https://github.com/Mozilla-Ocho/llamafile) 0.10.3 base carrying the opencoti patch series: advanced KV residency/quantization (PolyKV/TurboQuant/TCQ), rolling-KV window with host-RAM spill, DCA long-context extension, MTP speculative decode, sparse attention, RYS layer duplication, and a runtime introspection/control API. **Start with [USAGE.md](./USAGE.md)** — it explains exactly how this engine diverges from upstream llamafile, every added feature, its flags, defaults, limitations, and which features compose. This repo hosts the packaged release artifacts (they exceed GitHub's 2 GiB release-asset cap) plus the full patch series under [`patches/`](./patches) and the project documentation under [`docs/`](./docs). Release notes and per-release `SHA256SUMS` live under [`releases/`](./releases) and are mirrored on the corresponding GitHub release at [mann1x/opencoti](https://github.com/mann1x/opencoti/releases). ## Supported / target model families This engine is **not model-agnostic in what it optimizes**. Upstream llama.cpp GGUF support is unchanged (any GGUF that loads upstream loads here), but the opencoti feature set is built, tuned, and validated on two families: | Family | Role | Models | What's tuned for them | |---|---|---|---| | **Gemma-4** | **PRIMARY target** | 26B-A4B-128e MoE ("A4B"), dense 12B / 31B, elastic E-series (E2B/E4B) | head_dim 256/512 kernels, iSWA dual-cache (rolling-KV, SharedKVPool, DCA wiring), MTP via [gemma4-assistant drafters](https://huggingface.co/ManniX-ITA), fused MoE | | **Qwen** | SECONDARY / verification | Qwen3.5 / 3.6 dense + MoE + hybrid (gated-delta-net), Qwen2.5-14B-1M | head_dim 128 kernels, NextN self-spec MTP (fused multi-step draft), DCA long-context on 1M-class models | Other architectures run with upstream behavior; opencoti features either fall back safely or are unvalidated on them — see the model-families section at the top of [USAGE.md](./USAGE.md) before relying on an opencoti feature elsewhere. ## Artifacts | Artifact | Platform | GPU | |---|---|---| | `opencoti-llamafile---x86_64.llamafile` | x86_64 + aarch64 (fat APE; Linux/macOS/BSD) | CUDA 13.2 **embedded**, x86_64-linux (sm_75/80/86/89/90/120f) | | `opencoti-llamafile---win-x86_64.llamafile.exe` | same fat APE, incl. **Windows** | none embedded — CPU out of the box, GPU via side-load (below) | | `opencoti-llamafile---aarch64.llamafile` | same fat APE | CUDA 13.2 **embedded**, sbsa/aarch64-linux (sm_110f DGX Spark GB10 / Jetson Thor, sm_121a) | | `dso/-/ggml-cuda-x86_64.so` | side-load DSO for the `-win`/bare APE on x86_64-linux | CUDA 13.2, x86_64-linux (sm_75/80/86/89/90/120f) | | `dso/-/ggml-cuda-sbsa-aarch64.so` | side-load DSO for the `-win`/bare APE on aarch64-linux | CUDA 13.2, sbsa/aarch64-linux (sm_110f, sm_121a) | The APE host binary is identical across variants and runs natively on both x86_64 and aarch64. The variants differ ONLY in the embedded `ggml-cuda.so`: the full x86_64 artifact carries the x86_64-linux DSO (4.8 GB); the `-aarch64` artifact carries the sbsa/aarch64-linux DSO (1.8 GB — GPU out of the box on DGX Spark-class hosts); the `-win` variant carries none, because **Windows refuses to run executables larger than 4 GB** — it is the same APE renamed `.exe` (required on Windows) at ~55 MB. With no matching DSO available, inference falls back to CPU. ## Usage ```bash chmod +x opencoti-llamafile-*.llamafile # server mode sh ./opencoti-llamafile-*.llamafile --server --port 8080 \ -m .gguf -ngl 99 --flash-attn on # without --server it starts the interactive chat CLI ``` On the first GPU run the embedded CUDA DSO self-extracts to `~/.llamafile/v//`. `--version` prints the opencoti version string; `GET /props` exposes an `opencoti` introspection section at runtime (see USAGE.md §introspection). ### `-win` variant / GPU side-load - **Windows:** run `opencoti-llamafile---win-x86_64.llamafile.exe` directly (CPU inference works out of the box). For GPU, install the NVIDIA CUDA Toolkit + MSVC; on first `-ngl` run llamafile compiles a native `ggml-cuda.dll` on the fly into `%USERPROFILE%\.llamafile\`. - **Linux x86_64 (bare APE + GPU without the 4.8 GB download’s re-download):** drop the published side-load DSO where the engine looks before extracting an embedded one: ```bash mkdir -p ~/.llamafile/v// # e.g. 0.10.3 cp ggml-cuda-x86_64.so ~/.llamafile/v//ggml-cuda.so ``` Verify against `releases//SHA256SUMS` first. The `-win` file is still a normal APE — `sh ./…​.exe` runs fine on Linux/macOS/BSD. ## Verification Each artifact ships with a `MANIFEST.json` recording the artifact sha256, versionString, gitCommit, the full patch list, and the sha256 of every embedded backend DSO; `releases//SHA256SUMS` is the committed contract (CI verifies this repo's LFS blobs against it on every release tag). Verify the embedded DSO without running: ```bash unzip -p opencoti-llamafile-*.llamafile ggml-cuda.so | sha256sum ``` ## License Apache-2.0 (llamafile) + MIT (llama.cpp and bundled projects). opencoti patches and packaging are MIT.