README.md

metadata

license: apache-2.0
tags:
  - llamafile
  - llama.cpp
  - gguf
  - inference-engine
  - cuda
  - opencoti

opencoti-llamafile

Self-contained single-file inference engine from the opencoti project — a Mozilla-Ocho llamafile 0.10.3 base carrying the opencoti patch series: advanced KV residency/quantization (PolyKV/TurboQuant/TCQ), rolling-KV window with host-RAM spill, DCA long-context extension, MTP speculative decode, sparse attention, RYS layer duplication, and a runtime introspection/control API.

Start with USAGE.md — it explains exactly how this engine diverges from upstream llamafile, every added feature, its flags, defaults, limitations, and which features compose.

This repo hosts the packaged release artifacts (they exceed GitHub's 2 GiB release-asset cap) plus the full patch series under patches/ and the project documentation under docs/. Release notes and per-release SHA256SUMS live under releases/ and are mirrored on the corresponding GitHub release at mann1x/opencoti.

Supported / target model families

This engine is not model-agnostic in what it optimizes. Upstream llama.cpp GGUF support is unchanged (any GGUF that loads upstream loads here), but the opencoti feature set is built, tuned, and validated on two families:

Family	Role	Models	What's tuned for them
Gemma-4	PRIMARY target	26B-A4B-128e MoE ("A4B"), dense 12B / 31B, elastic E-series (E2B/E4B)	head_dim 256/512 kernels, iSWA dual-cache (rolling-KV, SharedKVPool, DCA wiring), MTP via gemma4-assistant drafters, fused MoE
Qwen	SECONDARY / verification	Qwen3.5 / 3.6 dense + MoE + hybrid (gated-delta-net), Qwen2.5-14B-1M	head_dim 128 kernels, NextN self-spec MTP (fused multi-step draft), DCA long-context on 1M-class models

Other architectures run with upstream behavior; opencoti features either fall back safely or are unvalidated on them — see the model-families section at the top of USAGE.md before relying on an opencoti feature elsewhere.

Artifacts

Artifact	Platform	GPU
opencoti-llamafile-<ver>-<tag>-x86_64.llamafile	x86_64 + aarch64 (fat APE; Linux/macOS/BSD)	CUDA 13.3 embedded, x86_64-linux (sm_75/80/86/89/90/120f)
opencoti-llamafile-<ver>-<tag>-win-x86_64.llamafile.exe	same fat APE, incl. Windows	none embedded — CPU out of the box, GPU via side-load (below)
opencoti-llamafile-<ver>-<tag>-aarch64.llamafile	same fat APE	CUDA 13.3 embedded, sbsa/aarch64-linux (sm_110f DGX Spark GB10 / Jetson Thor, sm_121a)
dso/<ver>-<tag>/ggml-cuda-x86_64.so	side-load DSO for the -win/bare APE on x86_64-linux	CUDA 13.3, x86_64-linux (sm_75/80/86/89/90/120f)
dso/<ver>-<tag>/ggml-cuda-sbsa-aarch64.so	side-load DSO for the -win/bare APE on aarch64-linux	CUDA 13.3, sbsa/aarch64-linux (sm_110f, sm_121a)

The APE host binary is identical across variants and runs natively on both x86_64 and aarch64. The variants differ ONLY in the embedded ggml-cuda.so: the full x86_64 artifact carries the x86_64-linux DSO (4.8 GB); the -aarch64 artifact carries the sbsa/aarch64-linux DSO (1.8 GB — GPU out of the box on DGX Spark-class hosts); the -win variant carries none, because Windows refuses to run executables larger than 4 GB — it is the same APE renamed .exe (required on Windows) at ~55 MB. With no matching DSO available, inference falls back to CPU.

Usage

chmod +x opencoti-llamafile-*.llamafile
# server mode
sh ./opencoti-llamafile-*.llamafile --server --port 8080 \
    -m <model>.gguf -ngl 99 --flash-attn on
# without --server it starts the interactive chat CLI

On the first GPU run the embedded CUDA DSO self-extracts to ~/.llamafile/v/<ver>/. --version prints the opencoti version string; GET /props exposes an opencoti introspection section at runtime (see USAGE.md §introspection).

-win variant / GPU side-load

• Windows: run opencoti-llamafile-<ver>-<tag>-win-x86_64.llamafile.exe directly (CPU inference works out of the box). For GPU, install the NVIDIA CUDA Toolkit + MSVC; on first -ngl run llamafile compiles a native ggml-cuda.dll on the fly into %USERPROFILE%\.llamafile\.

• Linux x86_64 (bare APE + GPU without the 4.8 GB download’s re-download): drop the published side-load DSO where the engine looks before extracting an embedded one:

  mkdir -p ~/.llamafile/v/<ver>/   # e.g. 0.10.3
  cp ggml-cuda-x86_64.so ~/.llamafile/v/<ver>/ggml-cuda.so
  

  Verify against releases/<tag>/SHA256SUMS first. The -win file is still a normal APE — sh ./…​.exe runs fine on Linux/macOS/BSD.

Verification

Each artifact ships with a MANIFEST.json recording the artifact sha256, versionString, gitCommit, the full patch list, and the sha256 of every embedded backend DSO; releases/<tag>/SHA256SUMS is the committed contract (CI verifies this repo's LFS blobs against it on every release tag). Verify the embedded DSO without running:

unzip -p opencoti-llamafile-*.llamafile ggml-cuda.so | sha256sum

License

Apache-2.0 (llamafile) + MIT (llama.cpp and bundled projects). opencoti patches and packaging are MIT.