docs: PolyKV stress benchmark section (#742)
Browse files
README.md
CHANGED
|
@@ -119,6 +119,33 @@ unzip -p opencoti-llamafile-*.llamafile ggml-cuda.so | sha256sum
|
|
| 119 |
Apache-2.0 (llamafile) + MIT (llama.cpp and bundled projects).
|
| 120 |
opencoti patches and packaging are MIT.
|
| 121 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 122 |
---
|
| 123 |
|
| 124 |
<img src="vodafone-logo.png" alt="Vodafone" width="96"/>
|
|
|
|
| 119 |
Apache-2.0 (llamafile) + MIT (llama.cpp and bundled projects).
|
| 120 |
opencoti patches and packaging are MIT.
|
| 121 |
|
| 122 |
+
|
| 123 |
+
## PolyKV multi-session stress benchmark (package_courier)
|
| 124 |
+
|
| 125 |
+
An agentic multi-session benchmark: LangGraph courier agents (tool-calling,
|
| 126 |
+
10-step delivery chains, 100 packages) run against ONE engine at
|
| 127 |
+
`--parallel 32`, ctx 163840, q8_0 KV, with a tps-floor scheduler that admits
|
| 128 |
+
new sessions only while every live session keeps ≥ floor tok/s (PolyKV
|
| 129 |
+
`/polykv/tps` SSE telemetry drives admission). Harness source:
|
| 130 |
+
`opencoti/langgraph/bench/package_courier/` in the repo; reports under
|
| 131 |
+
[`bench/reports/`](bench/reports/) (per-run HTML + raw stats).
|
| 132 |
+
|
| 133 |
+
| model (MTP) | floor | score | wall | delivery mean | fan-out |
|
| 134 |
+
|---|---|---|---|---|---|
|
| 135 |
+
| Qwen3.6-27B-Omnimerge-v4 Q6_K (NextN self-spec) | 15 | 99/100 | 2171 s | 146 s | ~7 |
|
| 136 |
+
| ″ | 5 | 99/100 | 2509 s | 467 s | →22 |
|
| 137 |
+
| ″ | 1 | 98/100 | 2803 s | 818 s | 32 (max) |
|
| 138 |
+
| Gemma-4 26B-A4B-it Q4_K_M (assistant-MTP) | 15 | 100/100 | 864 s | 51 s | med 6 |
|
| 139 |
+
| ″ | 5 | 99/100 | 1090 s | 145 s | med 14 |
|
| 140 |
+
| ″ | 1 | 95/100 | 1340 s | 346 s | 32 (max) |
|
| 141 |
+
|
| 142 |
+
Aggregate engine throughput stays pinned (Qwen ~75-80 tok/s regardless of
|
| 143 |
+
concurrency), so equilibrium workers ≈ aggregate/floor. The A4B + assistant-MTP
|
| 144 |
+
pair is ~2.5× faster end-to-end at every rung. All losses across all runs are
|
| 145 |
+
agent-side (turn-budget wander on hard packages) — zero tool errors, zero LLM
|
| 146 |
+
errors, engines clean throughout. The `ab-pooled` vs `ab-naive` pair isolates
|
| 147 |
+
the SharedKVPool win at identical workload.
|
| 148 |
+
|
| 149 |
---
|
| 150 |
|
| 151 |
<img src="vodafone-logo.png" alt="Vodafone" width="96"/>
|