sync: latest measured numbers
Browse files- RESULTS.md +18 -0
RESULTS.md
CHANGED
|
@@ -114,3 +114,21 @@ Scope caveat: a 44-request short-form matrix rules out gross corruption, not rar
|
|
| 114 |
corruption; the #36885 sentinel bug is claimed load-dependent. Methodology note: our first run
|
| 115 |
"failed" 7/44 — every failure was reasoning-narration truncated by an 80-token cap, zero wrong
|
| 116 |
values; budget raised to 400 and the checker's false accusations vanished. Check your checker.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 114 |
corruption; the #36885 sentinel bug is claimed load-dependent. Methodology note: our first run
|
| 115 |
"failed" 7/44 — every failure was reasoning-narration truncated by an 80-token cap, zero wrong
|
| 116 |
values; budget raised to 400 and the checker's false accusations vanished. Check your checker.
|
| 117 |
+
|
| 118 |
+
## SHOOTOUT round 2 (2026-08-28 late) — EXL3 lane rev 1df71c1 (MiaAI-Lab's concurrency+context update)
|
| 119 |
+
Re-ran after MiaAI-Lab shipped 7 commits (DFlash2/MLA KV page-sharing, 1M default context,
|
| 120 |
+
prefix-cache fix). Same unified protocol, our rig, their new image. What changed:
|
| 121 |
+
| metric | EXL3 v1 (bd7f55e) | EXL3 v2 (1df71c1) | SGLang fp8 (ours) |
|
| 122 |
+
|---|---:|---:|---:|
|
| 123 |
+
| code c1 | 51.9 | 45.3 | 29.3 |
|
| 124 |
+
| structured | 51.3 | 51.1 | 43.3 |
|
| 125 |
+
| prose | 23.7 | 25.0 | 29.2 |
|
| 126 |
+
| c8 aggregate | 63.8 | 57.7 | 78.1 |
|
| 127 |
+
| c12 aggregate | 53.3 | 59.9 | 83.5 |
|
| 128 |
+
| 54k-token prefill | (not tested) | **PASS** (worker survived, 119GB peak) | **worker dies ~62k** |
|
| 129 |
+
Read: her concurrency fix is real but partial — c12 improved +12% (53.3→59.9) yet still trails
|
| 130 |
+
our 83.5 and still degrades past c4. Her decisive new edge is LONG CONTEXT reliability: her lane
|
| 131 |
+
recalled a needle at 34k AND 54k tokens with the worker surviving, where our stack silently kills
|
| 132 |
+
rank1 at ~62k (the blocker from our night-3 probe). Her MLA/DFlash2 KV page-sharing is doing what
|
| 133 |
+
her commits claim. Honest scorecard now: EXL3 wins solo code + long context; SGLang fp8 wins prose
|
| 134 |
+
+ fleet concurrency. Both re-measured same rig, same night. Credit MiaAI-Lab for a fast, real fix.
|