randomllama commited on
Commit
da788b0
·
verified ·
1 Parent(s): 2f5385e

sync: latest measured numbers

Browse files
Files changed (1) hide show
  1. LADDER.md +11 -0
LADDER.md CHANGED
@@ -166,3 +166,14 @@ stream's start delay + decode; at c4 the fixed ramp amortizes over half the toke
166
  reads as a "dip". Unchanged conclusions: the engine's decode is monotone in batch size, and
167
  low-concurrency wall-clock aggregates understate steady-state engine throughput on BOTH stacks.
168
  Structured re-measure same session: 48.8-49.7 tok/s (x3) vs 43.3 earlier — run-to-run band.
 
 
 
 
 
 
 
 
 
 
 
 
166
  reads as a "dip". Unchanged conclusions: the engine's decode is monotone in batch size, and
167
  low-concurrency wall-clock aggregates understate steady-state engine throughput on BOTH stacks.
168
  Structured re-measure same session: 48.8-49.7 tok/s (x3) vs 43.3 earlier — run-to-run band.
169
+
170
+ ## Night queues 2+3 (2026-08-28 evening) — four rungs, one discovery, one blocker
171
+ | rung | change | verdict |
172
+ |---|---|---|
173
+ | N1 | num-continuous-decode-steps 3 | NULL — exact baseline; scheduler loop is not the 40ms floor |
174
+ | N2 | torch.compile | BOOT-FAIL — cuda-graph capture OOM at fraction 0.92; retry would need capped graph set + lower fraction, plus a compile tax on every boot. Not production-shaped |
175
+ | N3 | context 262k + fraction 0.93 | Speed-neutral (28.7 code, 80.2 c12), pool fully funded at 262,144 — BUT the 64k depth probe KILLED the worker (silent rank1 death mid-prefill, no traceback, docker state incoherent). Same class as the vLLM-route long-prefill worker-kill; falsifies "fp8 KV unaffected". Threshold between 32k (passes) and 62k (kills). >32k prompts UNVERIFIED on all configs until bisected. NOT promoted |
176
+ | N4 | enable-mixed-chunk | NULL on this protocol — short-prompt sweep cannot exercise it; needs a prefill-interference test before final verdict |
177
+ Conclusion of the verify-cost hunt: both cheap levers dead; the code-speed gap vs EXL3 is
178
+ quant-level. Next real levers: decoupled drafter (architecture), and the 64k crash bisect
179
+ (reliability before capacity). Production remains FP8T8X.