randomllama commited on
Commit
1cc17a5
·
verified ·
1 Parent(s): d2c1c38

sync: latest measured numbers

Browse files
Files changed (1) hide show
  1. LADDER.md +15 -9
LADDER.md CHANGED
@@ -215,12 +215,18 @@ prompts. Promotion gated on 19x21 + a 44-request correctness matrix under load:
215
  (c1 8/8, c4 12/12, c8 24/24). 2048 is the right choice only if long-context latency matters
216
  more than 8-way throughput.
217
 
218
- ### Mechanism: NOT established (negative result)
219
- Our dense-`[Σq × Σk/4]`-fp32-indexer-logits hypothesis did NOT survive its own test. We wired the
220
- unused `_should_chunk_mqa_logits` guard into the kpool path (verified live in-container via class
221
- MRO) and re-ran the failing case at chunk 8192: **still timed out at 300 s**, same as stock. So
222
- either the guard never fired at our free-memory level, or logits chunking is not the fix. A
223
- forced-unconditional-chunking diagnostic build is running to separate those. Until it reports,
224
- the honest statement is: **chunk size demonstrably controls whether 100k prompts complete on
225
- GB10; why, is unresolved.** Separately, the guard being defined-but-never-called on the kpool
226
- path is still a real upstream defect worth fixing regardless of whether it is our cause.
 
 
 
 
 
 
 
215
  (c1 8/8, c4 12/12, c8 24/24). 2048 is the right choice only if long-context latency matters
216
  more than 8-way throughput.
217
 
218
+ ### Mechanism: CONFIRMED (after one failed attempt)
219
+ Three builds, same 102,644-token unique prompt, chunk 8192, JIT pre-warmed, within one hour:
220
+ | build | result |
221
+ |---|---|
222
+ | stock | timeout at 300 s (x2) |
223
+ | stock + guard wired into kpool path (live-verified via container class MRO) | timeout at 300 s |
224
+ | same + logits chunked unconditionally at 512 rows (diagnostic build) | **PASS 135.0 s** |
225
+ So the dense `[Σq × Σk/4]` fp32 logits block IS the cause — chunking it, and nothing else, fixes
226
+ the case. The middle row is explained by the guard's own log line, which the diagnostic build
227
+ emitted 143 times: `num_q=4340 num_k=25664 guard_said_chunk=True budget=217873612`. The guard
228
+ correctly says "chunk" but its ~218 MB budget yields ~2,122-row chunks (25,664 x 4 bytes/row),
229
+ which is still too coarse here; 512-row chunks work. **Two defects, not one: the guard is never
230
+ called on the kpool path, AND its default budget is ~4x too generous for GB10 unified memory.**
231
+ Reported upstream on #36941. Note the first version of this section said the mechanism was NOT
232
+ established — that was written between the second and third builds and is kept in git history.