sync: latest measured numbers
Browse files
LADDER.md
CHANGED
|
@@ -166,3 +166,14 @@ stream's start delay + decode; at c4 the fixed ramp amortizes over half the toke
|
|
| 166 |
reads as a "dip". Unchanged conclusions: the engine's decode is monotone in batch size, and
|
| 167 |
low-concurrency wall-clock aggregates understate steady-state engine throughput on BOTH stacks.
|
| 168 |
Structured re-measure same session: 48.8-49.7 tok/s (x3) vs 43.3 earlier — run-to-run band.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 166 |
reads as a "dip". Unchanged conclusions: the engine's decode is monotone in batch size, and
|
| 167 |
low-concurrency wall-clock aggregates understate steady-state engine throughput on BOTH stacks.
|
| 168 |
Structured re-measure same session: 48.8-49.7 tok/s (x3) vs 43.3 earlier — run-to-run band.
|
| 169 |
+
|
| 170 |
+
## Night queues 2+3 (2026-08-28 evening) — four rungs, one discovery, one blocker
|
| 171 |
+
| rung | change | verdict |
|
| 172 |
+
|---|---|---|
|
| 173 |
+
| N1 | num-continuous-decode-steps 3 | NULL — exact baseline; scheduler loop is not the 40ms floor |
|
| 174 |
+
| N2 | torch.compile | BOOT-FAIL — cuda-graph capture OOM at fraction 0.92; retry would need capped graph set + lower fraction, plus a compile tax on every boot. Not production-shaped |
|
| 175 |
+
| N3 | context 262k + fraction 0.93 | Speed-neutral (28.7 code, 80.2 c12), pool fully funded at 262,144 — BUT the 64k depth probe KILLED the worker (silent rank1 death mid-prefill, no traceback, docker state incoherent). Same class as the vLLM-route long-prefill worker-kill; falsifies "fp8 KV unaffected". Threshold between 32k (passes) and 62k (kills). >32k prompts UNVERIFIED on all configs until bisected. NOT promoted |
|
| 176 |
+
| N4 | enable-mixed-chunk | NULL on this protocol — short-prompt sweep cannot exercise it; needs a prefill-interference test before final verdict |
|
| 177 |
+
Conclusion of the verify-cost hunt: both cheap levers dead; the code-speed gap vs EXL3 is
|
| 178 |
+
quant-level. Next real levers: decoupled drafter (architecture), and the 64k crash bisect
|
| 179 |
+
(reliability before capacity). Production remains FP8T8X.
|