sync: latest measured numbers
Browse files- RESULTS.md +17 -0
RESULTS.md
CHANGED
|
@@ -56,3 +56,20 @@ returned 2,462 chars of pure answer in the same wall time. For agent/tool worklo
|
|
| 56 |
disable thinking per-request ("chat_template_kwargs": {"enable_thinking": false}) or
|
| 57 |
budget max_tokens for reasoning + answer. Conditions: merge-sorted-lists code prompt,
|
| 58 |
600 max_tokens, temp 0, FP8T8V config.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 56 |
disable thinking per-request ("chat_template_kwargs": {"enable_thinking": false}) or
|
| 57 |
budget max_tokens for reasoning + answer. Conditions: merge-sorted-lists code prompt,
|
| 58 |
600 max_tokens, temp 0, FP8T8V config.
|
| 59 |
+
|
| 60 |
+
## Cross-stack comparison vs EXL3+vLLM (2026-08-28, same prompts, our hardware for our column)
|
| 61 |
+
MiaAI-Lab published GLM-5.3-Flash-EXL3-2x-DGX-Sparks (EXL3/TR3 4bpw by brandonmusic, custom
|
| 62 |
+
vLLM image, DFlash2 k=7, fp8_ds_mla KV, 900k context). Their headline 62.9/103.3/146.5 is the
|
| 63 |
+
"Structured" bench — counting 1 to 200 — a ~0.92-accept regime their own fine print separates
|
| 64 |
+
from prose (26.9) and long-context (24-27). We ran their exact protocol on our stack
|
| 65 |
+
(temp 0, thinking off, 400 max_tokens, top_p 1, warmed, n=5 medians, FP8T8V config):
|
| 66 |
+
| workload (their prompts) | EXL3+vLLM (their lab numbers) | ours (SGLang fp8-KV) |
|
| 67 |
+
|---|---:|---:|
|
| 68 |
+
| structured count-to-200 | 61.7 | 43.3 |
|
| 69 |
+
| prose hash-map | 26.9 | **29.2** |
|
| 70 |
+
Structured gap decomposes: they draft k=7 (max 8 tok/step, realize 6.43); we run D=5 (max 6,
|
| 71 |
+
realize ~5.9 — pinned at ceiling on this workload). D is workload-tunable; we have not re-swept
|
| 72 |
+
D for high-accept regimes. On the prose workload the SGLang stack is faster. Their genuine
|
| 73 |
+
edges, acknowledged: (1) weights quality — independent KLD panel puts EXL3 4bpw at ~official-FP8
|
| 74 |
+
level while NVFP4 (which we serve) scores 2.5x worse; (2) KV pool — 982k tokens vs our 84k
|
| 75 |
+
(context expansion on our stack is config work, queued). Credit: MiaAI-Lab and brandonmusic.
|