randomllama commited on
Commit
d2c1c38
·
verified ·
1 Parent(s): 18a14c3

sync: latest measured numbers

Browse files
Files changed (1) hide show
  1. LADDER.md +23 -0
LADDER.md CHANGED
@@ -201,3 +201,26 @@ fast at every size measured and is the difference between completing and not com
201
  ~100k tokens. Mechanism (dense `[Σq × Σk/4]` fp32 indexer logits scaling with TOTAL prompt length,
202
  with `_should_chunk_mqa_logits` defined but never called on the kpool path) and our public
203
  correction: sglang issue #36941.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
201
  ~100k tokens. Mechanism (dense `[Σq × Σk/4]` fp32 indexer logits scaling with TOTAL prompt length,
202
  with `_should_chunk_mqa_logits` defined but never called on the kpool path) and our public
203
  correction: sglang issue #36941.
204
+
205
+ ## Chunked-prefill size sweep (2026-08-29) — new production config
206
+ Single variable, everything else held at production settings. 102k probe = unique uncached
207
+ prompt, JIT pre-warmed, `POST /flush_cache` first.
208
+ | chunk | code | prose | c8 | c12 | 102,644-token prompt |
209
+ |---|---:|---:|---:|---:|---|
210
+ | 8192 (old prod) | 27.0-29.3 | 23.8 | 78.1 | 83.5 | **timeout at 300 s (reproduced twice)** |
211
+ | **4096 (NEW PROD)** | **28.6** | 23.6 | **77.4** | **83.2** | **PASS 104.0 s** |
212
+ | 2048 | 29.3 | 23.3 | 73.5 (~6% below band) | 82.4 | PASS 65.5 s |
213
+ **Production = `start-LC4.sh`** (chunk 4096): keeps the concurrency band AND completes 100k+
214
+ prompts. Promotion gated on 19x21 + a 44-request correctness matrix under load: 44/44 clean
215
+ (c1 8/8, c4 12/12, c8 24/24). 2048 is the right choice only if long-context latency matters
216
+ more than 8-way throughput.
217
+
218
+ ### Mechanism: NOT established (negative result)
219
+ Our dense-`[Σq × Σk/4]`-fp32-indexer-logits hypothesis did NOT survive its own test. We wired the
220
+ unused `_should_chunk_mqa_logits` guard into the kpool path (verified live in-container via class
221
+ MRO) and re-ran the failing case at chunk 8192: **still timed out at 300 s**, same as stock. So
222
+ either the guard never fired at our free-memory level, or logits chunking is not the fix. A
223
+ forced-unconditional-chunking diagnostic build is running to separate those. Until it reports,
224
+ the honest statement is: **chunk size demonstrably controls whether 100k prompts complete on
225
+ GB10; why, is unresolved.** Separately, the guard being defined-but-never-called on the kpool
226
+ path is still a real upstream defect worth fixing regardless of whether it is our cause.