sync: latest measured numbers
Browse files
LADDER.md
CHANGED
|
@@ -215,12 +215,18 @@ prompts. Promotion gated on 19x21 + a 44-request correctness matrix under load:
|
|
| 215 |
(c1 8/8, c4 12/12, c8 24/24). 2048 is the right choice only if long-context latency matters
|
| 216 |
more than 8-way throughput.
|
| 217 |
|
| 218 |
-
### Mechanism:
|
| 219 |
-
|
| 220 |
-
|
| 221 |
-
|
| 222 |
-
|
| 223 |
-
|
| 224 |
-
|
| 225 |
-
|
| 226 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 215 |
(c1 8/8, c4 12/12, c8 24/24). 2048 is the right choice only if long-context latency matters
|
| 216 |
more than 8-way throughput.
|
| 217 |
|
| 218 |
+
### Mechanism: CONFIRMED (after one failed attempt)
|
| 219 |
+
Three builds, same 102,644-token unique prompt, chunk 8192, JIT pre-warmed, within one hour:
|
| 220 |
+
| build | result |
|
| 221 |
+
|---|---|
|
| 222 |
+
| stock | timeout at 300 s (x2) |
|
| 223 |
+
| stock + guard wired into kpool path (live-verified via container class MRO) | timeout at 300 s |
|
| 224 |
+
| same + logits chunked unconditionally at 512 rows (diagnostic build) | **PASS 135.0 s** |
|
| 225 |
+
So the dense `[Σq × Σk/4]` fp32 logits block IS the cause — chunking it, and nothing else, fixes
|
| 226 |
+
the case. The middle row is explained by the guard's own log line, which the diagnostic build
|
| 227 |
+
emitted 143 times: `num_q=4340 num_k=25664 guard_said_chunk=True budget=217873612`. The guard
|
| 228 |
+
correctly says "chunk" but its ~218 MB budget yields ~2,122-row chunks (25,664 x 4 bytes/row),
|
| 229 |
+
which is still too coarse here; 512-row chunks work. **Two defects, not one: the guard is never
|
| 230 |
+
called on the kpool path, AND its default budget is ~4x too generous for GB10 unified memory.**
|
| 231 |
+
Reported upstream on #36941. Note the first version of this section said the mechanism was NOT
|
| 232 |
+
established — that was written between the second and third builds and is kept in git history.
|