docs: -win variant + side-load DSO + target model families
Browse files
README.md
CHANGED
|
@@ -32,16 +32,39 @@ under [`releases/`](./releases) and are mirrored on the corresponding
|
|
| 32 |
GitHub release at
|
| 33 |
[mann1x/opencoti](https://github.com/mann1x/opencoti/releases).
|
| 34 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 35 |
## Artifacts
|
| 36 |
|
| 37 |
| Artifact | Platform | GPU |
|
| 38 |
|---|---|---|
|
| 39 |
-
| `opencoti-llamafile-<ver>-<tag>-x86_64.llamafile` | x86_64 + aarch64 (fat APE; Linux/macOS/
|
|
|
|
|
|
|
| 40 |
| `opencoti-llamafile-<ver>-<tag>-aarch64.llamafile` *(planned)* | same fat APE | CUDA 13.2, sbsa/aarch64 (sm_110f Jetson Thor, sm_121a DGX Spark GB10) |
|
| 41 |
|
| 42 |
-
The APE host binary
|
| 43 |
-
|
| 44 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 45 |
|
| 46 |
## Usage
|
| 47 |
|
|
@@ -58,6 +81,24 @@ On the first GPU run the embedded CUDA DSO self-extracts to
|
|
| 58 |
string; `GET /props` exposes an `opencoti` introspection section at
|
| 59 |
runtime (see USAGE.md §introspection).
|
| 60 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 61 |
## Verification
|
| 62 |
|
| 63 |
Each artifact ships with a `MANIFEST.json` recording the artifact
|
|
|
|
| 32 |
GitHub release at
|
| 33 |
[mann1x/opencoti](https://github.com/mann1x/opencoti/releases).
|
| 34 |
|
| 35 |
+
## Supported / target model families
|
| 36 |
+
|
| 37 |
+
This engine is **not model-agnostic in what it optimizes**. Upstream
|
| 38 |
+
llama.cpp GGUF support is unchanged (any GGUF that loads upstream
|
| 39 |
+
loads here), but the opencoti feature set is built, tuned, and
|
| 40 |
+
validated on two families:
|
| 41 |
+
|
| 42 |
+
| Family | Role | Models | What's tuned for them |
|
| 43 |
+
|---|---|---|---|
|
| 44 |
+
| **Gemma-4** | **PRIMARY target** | 26B-A4B-128e MoE ("A4B"), dense 12B / 31B, elastic E-series (E2B/E4B) | head_dim 256/512 kernels, iSWA dual-cache (rolling-KV, SharedKVPool, DCA wiring), MTP via [gemma4-assistant drafters](https://huggingface.co/ManniX-ITA), fused MoE |
|
| 45 |
+
| **Qwen** | SECONDARY / verification | Qwen3.5 / 3.6 dense + MoE + hybrid (gated-delta-net), Qwen2.5-14B-1M | head_dim 128 kernels, NextN self-spec MTP (fused multi-step draft), DCA long-context on 1M-class models |
|
| 46 |
+
|
| 47 |
+
Other architectures run with upstream behavior; opencoti features
|
| 48 |
+
either fall back safely or are unvalidated on them — see the
|
| 49 |
+
model-families section at the top of [USAGE.md](./USAGE.md) before
|
| 50 |
+
relying on an opencoti feature elsewhere.
|
| 51 |
+
|
| 52 |
## Artifacts
|
| 53 |
|
| 54 |
| Artifact | Platform | GPU |
|
| 55 |
|---|---|---|
|
| 56 |
+
| `opencoti-llamafile-<ver>-<tag>-x86_64.llamafile` | x86_64 + aarch64 (fat APE; Linux/macOS/BSD) | CUDA 13.2 **embedded**, x86_64-linux (sm_75/80/86/89/90/120f) |
|
| 57 |
+
| `opencoti-llamafile-<ver>-<tag>-win-x86_64.llamafile.exe` | same fat APE, incl. **Windows** | none embedded — CPU out of the box, GPU via side-load (below) |
|
| 58 |
+
| `dso/<ver>-<tag>/ggml-cuda-x86_64.so` | side-load DSO for the `-win`/bare APE on x86_64-linux | CUDA 13.2, x86_64-linux (sm_75/80/86/89/90/120f) |
|
| 59 |
| `opencoti-llamafile-<ver>-<tag>-aarch64.llamafile` *(planned)* | same fat APE | CUDA 13.2, sbsa/aarch64 (sm_110f Jetson Thor, sm_121a DGX Spark GB10) |
|
| 60 |
|
| 61 |
+
The APE host binary is identical across variants and runs natively on
|
| 62 |
+
both x86_64 and aarch64. The variants differ ONLY in the embedded
|
| 63 |
+
`ggml-cuda.so`: the full x86_64 artifact carries it inside (4.8 GB);
|
| 64 |
+
the `-win` variant carries none, because **Windows refuses to run
|
| 65 |
+
executables larger than 4 GB** — it is the same APE renamed `.exe`
|
| 66 |
+
(required on Windows) at ~55 MB. With no matching DSO available,
|
| 67 |
+
inference falls back to CPU.
|
| 68 |
|
| 69 |
## Usage
|
| 70 |
|
|
|
|
| 81 |
string; `GET /props` exposes an `opencoti` introspection section at
|
| 82 |
runtime (see USAGE.md §introspection).
|
| 83 |
|
| 84 |
+
### `-win` variant / GPU side-load
|
| 85 |
+
|
| 86 |
+
- **Windows:** run `opencoti-llamafile-<ver>-<tag>-win-x86_64.llamafile.exe`
|
| 87 |
+
directly (CPU inference works out of the box). For GPU, install the
|
| 88 |
+
NVIDIA CUDA Toolkit + MSVC; on first `-ngl` run llamafile compiles a
|
| 89 |
+
native `ggml-cuda.dll` on the fly into `%USERPROFILE%\.llamafile\`.
|
| 90 |
+
- **Linux x86_64 (bare APE + GPU without the 4.8 GB download’s
|
| 91 |
+
re-download):** drop the published side-load DSO where the engine
|
| 92 |
+
looks before extracting an embedded one:
|
| 93 |
+
|
| 94 |
+
```bash
|
| 95 |
+
mkdir -p ~/.llamafile/v/<ver>/ # e.g. 0.10.3
|
| 96 |
+
cp ggml-cuda-x86_64.so ~/.llamafile/v/<ver>/ggml-cuda.so
|
| 97 |
+
```
|
| 98 |
+
|
| 99 |
+
Verify against `releases/<tag>/SHA256SUMS` first. The `-win` file is
|
| 100 |
+
still a normal APE — `sh ./….exe` runs fine on Linux/macOS/BSD.
|
| 101 |
+
|
| 102 |
## Verification
|
| 103 |
|
| 104 |
Each artifact ships with a `MANIFEST.json` recording the artifact
|