sync: latest measured numbers
Browse files
LADDER.md
CHANGED
|
@@ -201,3 +201,26 @@ fast at every size measured and is the difference between completing and not com
|
|
| 201 |
~100k tokens. Mechanism (dense `[Σq × Σk/4]` fp32 indexer logits scaling with TOTAL prompt length,
|
| 202 |
with `_should_chunk_mqa_logits` defined but never called on the kpool path) and our public
|
| 203 |
correction: sglang issue #36941.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 201 |
~100k tokens. Mechanism (dense `[Σq × Σk/4]` fp32 indexer logits scaling with TOTAL prompt length,
|
| 202 |
with `_should_chunk_mqa_logits` defined but never called on the kpool path) and our public
|
| 203 |
correction: sglang issue #36941.
|
| 204 |
+
|
| 205 |
+
## Chunked-prefill size sweep (2026-08-29) — new production config
|
| 206 |
+
Single variable, everything else held at production settings. 102k probe = unique uncached
|
| 207 |
+
prompt, JIT pre-warmed, `POST /flush_cache` first.
|
| 208 |
+
| chunk | code | prose | c8 | c12 | 102,644-token prompt |
|
| 209 |
+
|---|---:|---:|---:|---:|---|
|
| 210 |
+
| 8192 (old prod) | 27.0-29.3 | 23.8 | 78.1 | 83.5 | **timeout at 300 s (reproduced twice)** |
|
| 211 |
+
| **4096 (NEW PROD)** | **28.6** | 23.6 | **77.4** | **83.2** | **PASS 104.0 s** |
|
| 212 |
+
| 2048 | 29.3 | 23.3 | 73.5 (~6% below band) | 82.4 | PASS 65.5 s |
|
| 213 |
+
**Production = `start-LC4.sh`** (chunk 4096): keeps the concurrency band AND completes 100k+
|
| 214 |
+
prompts. Promotion gated on 19x21 + a 44-request correctness matrix under load: 44/44 clean
|
| 215 |
+
(c1 8/8, c4 12/12, c8 24/24). 2048 is the right choice only if long-context latency matters
|
| 216 |
+
more than 8-way throughput.
|
| 217 |
+
|
| 218 |
+
### Mechanism: NOT established (negative result)
|
| 219 |
+
Our dense-`[Σq × Σk/4]`-fp32-indexer-logits hypothesis did NOT survive its own test. We wired the
|
| 220 |
+
unused `_should_chunk_mqa_logits` guard into the kpool path (verified live in-container via class
|
| 221 |
+
MRO) and re-ran the failing case at chunk 8192: **still timed out at 300 s**, same as stock. So
|
| 222 |
+
either the guard never fired at our free-memory level, or logits chunking is not the fix. A
|
| 223 |
+
forced-unconditional-chunking diagnostic build is running to separate those. Until it reports,
|
| 224 |
+
the honest statement is: **chunk size demonstrably controls whether 100k prompts complete on
|
| 225 |
+
GB10; why, is unresolved.** Separately, the guard being defined-but-never-called on the kpool
|
| 226 |
+
path is still a real upstream defect worth fixing regardless of whether it is our cause.
|