README.md

---
license: apache-2.0
tags:
  - llamafile
  - llama.cpp
  - gguf
  - inference-engine
  - cuda
  - opencoti
---

# opencoti-llamafile

Self-contained single-file inference engine from the
[opencoti](https://github.com/mann1x/opencoti) project — a
[Mozilla-Ocho llamafile](https://github.com/Mozilla-Ocho/llamafile)
0.10.3 base carrying the opencoti patch series: advanced KV
residency/quantization (PolyKV/TurboQuant/TCQ), rolling-KV window with
host-RAM spill, DCA long-context extension, MTP speculative decode,
sparse attention, RYS layer duplication, and a runtime
introspection/control API.

**Start with [USAGE.md](./USAGE.md)** — it explains exactly how this
engine diverges from upstream llamafile, every added feature, its
flags, defaults, limitations, and which features compose.

This repo hosts the packaged release artifacts (they exceed GitHub's
2 GiB release-asset cap) plus the full patch series under
[`patches/`](./patches) and the project documentation under
[`docs/`](./docs). Release notes and per-release `SHA256SUMS` live
under [`releases/`](./releases) and are mirrored on the corresponding
GitHub release at
[mann1x/opencoti](https://github.com/mann1x/opencoti/releases).

## Supported / target model families

This engine is **not model-agnostic in what it optimizes**. Upstream
llama.cpp GGUF support is unchanged (any GGUF that loads upstream
loads here), but the opencoti feature set is built, tuned, and
validated on two families:

| Family | Role | Models | What's tuned for them |
|---|---|---|---|
| **Gemma-4** | **PRIMARY target** | 26B-A4B-128e MoE ("A4B"), dense 12B / 31B, elastic E-series (E2B/E4B) | head_dim 256/512 kernels, iSWA dual-cache (rolling-KV, SharedKVPool, DCA wiring), MTP via [gemma4-assistant drafters](https://huggingface.co/ManniX-ITA), fused MoE |
| **Qwen** | SECONDARY / verification | Qwen3.5 / 3.6 dense + MoE + hybrid (gated-delta-net), Qwen2.5-14B-1M | head_dim 128 kernels, NextN self-spec MTP (fused multi-step draft), DCA long-context on 1M-class models |

Other architectures run with upstream behavior; opencoti features
either fall back safely or are unvalidated on them — see the
model-families section at the top of [USAGE.md](./USAGE.md) before
relying on an opencoti feature elsewhere.

## Artifacts

| Artifact | Platform | GPU |
|---|---|---|
| `opencoti-llamafile-<ver>-<tag>-x86_64.llamafile` | x86_64 + aarch64 (fat APE; Linux/macOS/BSD) | CUDA 13.2 **embedded**, x86_64-linux (sm_75/80/86/89/90/120f) |
| `opencoti-llamafile-<ver>-<tag>-win-x86_64.llamafile.exe` | same fat APE, incl. **Windows** | none embedded — CPU out of the box, GPU via side-load (below) |
| `opencoti-llamafile-<ver>-<tag>-aarch64.llamafile` | same fat APE | CUDA 13.2 **embedded**, sbsa/aarch64-linux (sm_110f DGX Spark GB10 / Jetson Thor, sm_121a) |
| `dso/<ver>-<tag>/ggml-cuda-x86_64.so` | side-load DSO for the `-win`/bare APE on x86_64-linux | CUDA 13.2, x86_64-linux (sm_75/80/86/89/90/120f) |
| `dso/<ver>-<tag>/ggml-cuda-sbsa-aarch64.so` | side-load DSO for the `-win`/bare APE on aarch64-linux | CUDA 13.2, sbsa/aarch64-linux (sm_110f, sm_121a) |

The APE host binary is identical across variants and runs natively on
both x86_64 and aarch64. The variants differ ONLY in the embedded
`ggml-cuda.so`: the full x86_64 artifact carries the x86_64-linux DSO
(4.8 GB); the `-aarch64` artifact carries the sbsa/aarch64-linux DSO
(1.8 GB — GPU out of the box on DGX Spark-class hosts); the `-win`
variant carries none, because **Windows refuses to run executables
larger than 4 GB** — it is the same APE renamed `.exe` (required on
Windows) at ~55 MB. With no matching DSO available, inference falls
back to CPU.

## Usage

```bash
chmod +x opencoti-llamafile-*.llamafile
# server mode
sh ./opencoti-llamafile-*.llamafile --server --port 8080 \
    -m <model>.gguf -ngl 99 --flash-attn on
# without --server it starts the interactive chat CLI
```

On the first GPU run the embedded CUDA DSO self-extracts to
`~/.llamafile/v/<ver>/`. `--version` prints the opencoti version
string; `GET /props` exposes an `opencoti` introspection section at
runtime (see USAGE.md §introspection).

### `-win` variant / GPU side-load

- **Windows:** run `opencoti-llamafile-<ver>-<tag>-win-x86_64.llamafile.exe`
  directly (CPU inference works out of the box). For GPU, install the
  NVIDIA CUDA Toolkit + MSVC; on first `-ngl` run llamafile compiles a
  native `ggml-cuda.dll` on the fly into `%USERPROFILE%\.llamafile\`.
- **Linux x86_64 (bare APE + GPU without the 4.8 GB download’s
  re-download):** drop the published side-load DSO where the engine
  looks before extracting an embedded one:

  ```bash
  mkdir -p ~/.llamafile/v/<ver>/   # e.g. 0.10.3
  cp ggml-cuda-x86_64.so ~/.llamafile/v/<ver>/ggml-cuda.so
  ```

  Verify against `releases/<tag>/SHA256SUMS` first. The `-win` file is
  still a normal APE — `sh ./…​.exe` runs fine on Linux/macOS/BSD.

## Verification

Each artifact ships with a `MANIFEST.json` recording the artifact
sha256, versionString, gitCommit, the full patch list, and the sha256
of every embedded backend DSO; `releases/<tag>/SHA256SUMS` is the
committed contract (CI verifies this repo's LFS blobs against it on
every release tag). Verify the embedded DSO without running:

```bash
unzip -p opencoti-llamafile-*.llamafile ggml-cuda.so | sha256sum
```

## License

Apache-2.0 (llamafile) + MIT (llama.cpp and bundled projects).
opencoti patches and packaging are MIT.