ManniX-ITA commited on
Commit
24b668f
·
verified ·
1 Parent(s): 5da0e0e

docs: -win variant + side-load DSO + target model families

Browse files
Files changed (1) hide show
  1. README.md +45 -4
README.md CHANGED
@@ -32,16 +32,39 @@ under [`releases/`](./releases) and are mirrored on the corresponding
32
  GitHub release at
33
  [mann1x/opencoti](https://github.com/mann1x/opencoti/releases).
34
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
35
  ## Artifacts
36
 
37
  | Artifact | Platform | GPU |
38
  |---|---|---|
39
- | `opencoti-llamafile-<ver>-<tag>-x86_64.llamafile` | x86_64 + aarch64 (fat APE; Linux/macOS/Windows/BSD) | CUDA 13.2, x86_64-linux (sm_75/80/86/89/90/120f) |
 
 
40
  | `opencoti-llamafile-<ver>-<tag>-aarch64.llamafile` *(planned)* | same fat APE | CUDA 13.2, sbsa/aarch64 (sm_110f Jetson Thor, sm_121a DGX Spark GB10) |
41
 
42
- The APE host binary runs natively on both x86_64 and aarch64; the two
43
- artifacts differ only in the embedded `ggml-cuda.so`. On a platform
44
- with no matching DSO, inference falls back to CPU.
 
 
 
 
45
 
46
  ## Usage
47
 
@@ -58,6 +81,24 @@ On the first GPU run the embedded CUDA DSO self-extracts to
58
  string; `GET /props` exposes an `opencoti` introspection section at
59
  runtime (see USAGE.md §introspection).
60
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
61
  ## Verification
62
 
63
  Each artifact ships with a `MANIFEST.json` recording the artifact
 
32
  GitHub release at
33
  [mann1x/opencoti](https://github.com/mann1x/opencoti/releases).
34
 
35
+ ## Supported / target model families
36
+
37
+ This engine is **not model-agnostic in what it optimizes**. Upstream
38
+ llama.cpp GGUF support is unchanged (any GGUF that loads upstream
39
+ loads here), but the opencoti feature set is built, tuned, and
40
+ validated on two families:
41
+
42
+ | Family | Role | Models | What's tuned for them |
43
+ |---|---|---|---|
44
+ | **Gemma-4** | **PRIMARY target** | 26B-A4B-128e MoE ("A4B"), dense 12B / 31B, elastic E-series (E2B/E4B) | head_dim 256/512 kernels, iSWA dual-cache (rolling-KV, SharedKVPool, DCA wiring), MTP via [gemma4-assistant drafters](https://huggingface.co/ManniX-ITA), fused MoE |
45
+ | **Qwen** | SECONDARY / verification | Qwen3.5 / 3.6 dense + MoE + hybrid (gated-delta-net), Qwen2.5-14B-1M | head_dim 128 kernels, NextN self-spec MTP (fused multi-step draft), DCA long-context on 1M-class models |
46
+
47
+ Other architectures run with upstream behavior; opencoti features
48
+ either fall back safely or are unvalidated on them — see the
49
+ model-families section at the top of [USAGE.md](./USAGE.md) before
50
+ relying on an opencoti feature elsewhere.
51
+
52
  ## Artifacts
53
 
54
  | Artifact | Platform | GPU |
55
  |---|---|---|
56
+ | `opencoti-llamafile-<ver>-<tag>-x86_64.llamafile` | x86_64 + aarch64 (fat APE; Linux/macOS/BSD) | CUDA 13.2 **embedded**, x86_64-linux (sm_75/80/86/89/90/120f) |
57
+ | `opencoti-llamafile-<ver>-<tag>-win-x86_64.llamafile.exe` | same fat APE, incl. **Windows** | none embedded — CPU out of the box, GPU via side-load (below) |
58
+ | `dso/<ver>-<tag>/ggml-cuda-x86_64.so` | side-load DSO for the `-win`/bare APE on x86_64-linux | CUDA 13.2, x86_64-linux (sm_75/80/86/89/90/120f) |
59
  | `opencoti-llamafile-<ver>-<tag>-aarch64.llamafile` *(planned)* | same fat APE | CUDA 13.2, sbsa/aarch64 (sm_110f Jetson Thor, sm_121a DGX Spark GB10) |
60
 
61
+ The APE host binary is identical across variants and runs natively on
62
+ both x86_64 and aarch64. The variants differ ONLY in the embedded
63
+ `ggml-cuda.so`: the full x86_64 artifact carries it inside (4.8 GB);
64
+ the `-win` variant carries none, because **Windows refuses to run
65
+ executables larger than 4 GB** — it is the same APE renamed `.exe`
66
+ (required on Windows) at ~55 MB. With no matching DSO available,
67
+ inference falls back to CPU.
68
 
69
  ## Usage
70
 
 
81
  string; `GET /props` exposes an `opencoti` introspection section at
82
  runtime (see USAGE.md §introspection).
83
 
84
+ ### `-win` variant / GPU side-load
85
+
86
+ - **Windows:** run `opencoti-llamafile-<ver>-<tag>-win-x86_64.llamafile.exe`
87
+ directly (CPU inference works out of the box). For GPU, install the
88
+ NVIDIA CUDA Toolkit + MSVC; on first `-ngl` run llamafile compiles a
89
+ native `ggml-cuda.dll` on the fly into `%USERPROFILE%\.llamafile\`.
90
+ - **Linux x86_64 (bare APE + GPU without the 4.8 GB download’s
91
+ re-download):** drop the published side-load DSO where the engine
92
+ looks before extracting an embedded one:
93
+
94
+ ```bash
95
+ mkdir -p ~/.llamafile/v/<ver>/ # e.g. 0.10.3
96
+ cp ggml-cuda-x86_64.so ~/.llamafile/v/<ver>/ggml-cuda.so
97
+ ```
98
+
99
+ Verify against `releases/<tag>/SHA256SUMS` first. The `-win` file is
100
+ still a normal APE — `sh ./…​.exe` runs fine on Linux/macOS/BSD.
101
+
102
  ## Verification
103
 
104
  Each artifact ships with a `MANIFEST.json` recording the artifact