File size: 2,776 Bytes
f0fcdab | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 | ---
license: apache-2.0
tags:
- llamafile
- llama.cpp
- gguf
- inference-engine
- cuda
- opencoti
---
# opencoti-llamafile
Self-contained single-file inference engine from the
[opencoti](https://github.com/mann1x/opencoti) project — a
[Mozilla-Ocho llamafile](https://github.com/Mozilla-Ocho/llamafile)
0.10.3 base carrying the opencoti patch series: advanced KV
residency/quantization (PolyKV/TurboQuant/TCQ), rolling-KV window with
host-RAM spill, DCA long-context extension, MTP speculative decode,
sparse attention, RYS layer duplication, and a runtime
introspection/control API.
**Start with [USAGE.md](./USAGE.md)** — it explains exactly how this
engine diverges from upstream llamafile, every added feature, its
flags, defaults, limitations, and which features compose.
This repo hosts the packaged release artifacts (they exceed GitHub's
2 GiB release-asset cap) plus the full patch series under
[`patches/`](./patches) and the project documentation under
[`docs/`](./docs). Release notes and per-release `SHA256SUMS` live
under [`releases/`](./releases) and are mirrored on the corresponding
GitHub release at
[mann1x/opencoti](https://github.com/mann1x/opencoti/releases).
## Artifacts
| Artifact | Platform | GPU |
|---|---|---|
| `opencoti-llamafile-<ver>-<tag>-x86_64.llamafile` | x86_64 + aarch64 (fat APE; Linux/macOS/Windows/BSD) | CUDA 13.2, x86_64-linux (sm_75/80/86/89/90/120f) |
| `opencoti-llamafile-<ver>-<tag>-aarch64.llamafile` *(planned)* | same fat APE | CUDA 13.2, sbsa/aarch64 (sm_110f Jetson Thor, sm_121a DGX Spark GB10) |
The APE host binary runs natively on both x86_64 and aarch64; the two
artifacts differ only in the embedded `ggml-cuda.so`. On a platform
with no matching DSO, inference falls back to CPU.
## Usage
```bash
chmod +x opencoti-llamafile-*.llamafile
# server mode
sh ./opencoti-llamafile-*.llamafile --server --port 8080 \
-m <model>.gguf -ngl 99 --flash-attn on
# without --server it starts the interactive chat CLI
```
On the first GPU run the embedded CUDA DSO self-extracts to
`~/.llamafile/v/<ver>/`. `--version` prints the opencoti version
string; `GET /props` exposes an `opencoti` introspection section at
runtime (see USAGE.md §introspection).
## Verification
Each artifact ships with a `MANIFEST.json` recording the artifact
sha256, versionString, gitCommit, the full patch list, and the sha256
of every embedded backend DSO; `releases/<tag>/SHA256SUMS` is the
committed contract (CI verifies this repo's LFS blobs against it on
every release tag). Verify the embedded DSO without running:
```bash
unzip -p opencoti-llamafile-*.llamafile ggml-cuda.so | sha256sum
```
## License
Apache-2.0 (llamafile) + MIT (llama.cpp and bundled projects).
opencoti patches and packaging are MIT.
|