Restate speculative-decoding tables under the EOS-cut protocol; add real-world sampling numbers

README.md

@@ -17,7 +17,7 @@ tags:
 
 # Qwen3.8-27B-Alis-MLX-8bit
 
-![Build size vs decode speed across the 8-bit, 6-bit and 4-bit builds, with Korean corpus perplexity and both speculative paths, MTP k=2 and the DSpark drafter](assets/builds.png)
+![Build size vs decode speed across the 8-bit, 6-bit and 4-bit builds, with Korean corpus perplexity and the speculative ladder on the 4-bit build: MTP k=2 and the gated MTP k=4 operating point](assets/builds.png)
 
 An 8-bit MLX quantization of [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) for Apple silicon, with the **vision tower and the MTP head both preserved**. 27.9 GB on disk, 21.8 tok/s decode, and corpus perplexity **statistically indistinguishable from the bf16 original** on English, Korean, and code alike — paired over ≈103K scored tokens. This is the fidelity build of the set.
 
@@ -150,25 +150,33 @@ mlx_lm.generate --model avlp12/Qwen3.8-27B-Alis-MLX-8bit \
   --max-tokens 512
 ```
 
-**Every speculative figure below was measured on the 4-bit build of this set — its earlier uniform 4-bit weights — not on this one.** It is quoted here because it is what the fork's two speculative paths do on this checkpoint, and because the ordering between them is informative — but the multiples do not transfer to this build, for a reason spelled out under the table. Measured on one Apple M3 Ultra (512 GB), greedy, 240 tokens per prompt, timed after an 8-token warmup, every configuration in the same process, over four prompts: chat, code, math, and Korean.
+**Correction (2026-08-16).** The speculative table earlier revisions of this card quoted from the 4-bit build — plain 37.63, MTP k=2 50.36, DSpark 62.21 — came from a harness that **did not stop at end-of-sequence**: on prompts whose answer ended inside the fixed window, the remainder measured post-termination behavior, and on the math prompt the model spent that tail copying its own finished answer (acceptance 4.53), which carried most of the headline. Those figures are retracted, as the footnote they carried promised they would be if unreproducible. Everything below is the re-measurement: **the harness stops at EOS, the four prompts (chat / code / math / Korean) are long-form** so every scored token is genuine answer decode, sampled rows are medians of three runs per prompt averaged across prompts, and stop-detection stays out of the timed loop.
 
-| Configuration, 4-bit build | 4-prompt avg | 3 English prompts | chat | code | math | ko |
-|---|---|---|---|---|---|---|
-| plain | 37.63 | 37.64 | 37.7 | 37.6 | 37.6 | 37.6 |
-| MTP k=2 | 50.36 (1.34×) | 55.71 (1.48×) | 50.6 | 54.6 | 62.0 | 34.3 |
-| DSpark drafter | 62.21 (1.65×) | 71.86 (1.91×) | 49.0 | 75.1 | 91.5 | 33.3 |
+The headline tables are still **measured on the 4-bit build of this set** (its earlier uniform weights; the AWQ swap measures within ±1%, paired) — quoted because they are what the fork's two speculative paths do on this checkpoint. This build now also has its own EOS-clean measurement, further below. Greedy first:
 
-Lossless on that build: all four prompts reproduced plain greedy decoding **token for token through 80 tokens**.
+| Configuration (greedy, 4-bit build) | tok/s | vs plain |
+|---|---|---|
+| plain | 37.6 | 1.00 |
+| DSpark drafter (block 8) | 48.3 | 1.28× |
+| MTP k=2 | 46.8 | 1.24× |
+| **MTP k=4 + `min_draft_p` 0.6** | **52.8** | **1.40×** |
 
-**Three footnotes on this table, added after an audit of the measurement harness.** First, the harness did **not stop at end-of-sequence** — every run decoded a fixed 240 tokens, so on prompts whose answer ends early, part of the window measures post-termination behavior, and per-prompt figures are sensitive to what a build happens to do after its answer ends. Read the averages, not the per-prompt cells. Second, acceptance-rate comparisons **between builds** are only meaningful on a common token sequence, scored paired — each build free-running its own text measures the text as much as the build. Third, the 4-bit build measured here was that repo's earlier uniform release; the AWQ weights now hosted there measure **within ±1%** of it on both speculative paths under an EOS-cut paired protocol. A full re-measurement under the corrected harness is a separate track.
+And under the shipped sampling defaults — temperature 1.0, top_p 0.95, top_k 20, what a chat deployment actually runs — using truncated rejection sampling, which passes target and draft distributions through the same temperature/top-p/top-k chain and so **preserves the requested output distribution exactly** (total variation ≤ 0.0014 against a synthetic oracle; 160/160 token match in the greedy limit):
 
-**Why this build cannot inherit those multiples.** What lifted them is a small-M quantized-GEMM kernel that the fork now enables from inside `mlx_lm.utils.load()`. It fires only in the multi-token verification window (M = 6..8) and only on **4-bit, group-64** weights, so it never engages on an 8-bit build; plain decode at M = 1 is untouched on every build either way. The gap it fills is upstream: [ml-explore/mlx#4265](https://github.com/ml-explore/mlx/issues/4265). Set `MLXLM_NO_FAST_QMM=1` to opt out.
+| Configuration (temp 1.0 · top_p 0.95 · top_k 20, 4-bit build) | 240 tok | vs plain | 1024 tok | vs plain |
+|---|---|---|---|---|
+| plain, sampler on | 37.16 | 1.00 | 36.91 | 1.00 |
+| DSpark drafter | 41.94 | 1.13× | 39.95 | 1.08× |
+| MTP k=2 | 47.74 | 1.28× | 44.10 | 1.19× |
+| **MTP k=4 + `min_draft_p` 0.6** | **48.05** | **1.29×** | **45.13** | **1.22×** |
 
-On this build MTP was measured at **21.9 to 31.3 tok/s, a 43% gain** — but treat that as a **pre-kernel measurement on an older three-prompt English-and-code set**, not a row of the table above, and not re-measured since. k=2 remains the sensible starting point here; measure your own workload rather than borrowing a number from either source.
+The gated k=4 row is the 4-bit recommendation (`mtp_num_draft_tokens=4, min_draft_p=0.6` via the Python API, plus `mtp_spec_temp=1.0, mtp_spec_top_k=20, mtp_spec_top_p=0.95` under the sampling defaults — the gate is what makes k=4 win; ungated deeper k measured below k=2). The small-M quantized-GEMM kernel behind the 4-bit verification window fires only on **4-bit, group-64** weights ([ml-explore/mlx#4265](https://github.com/ml-explore/mlx/issues/4265); kill switch `MLXLM_NO_FAST_QMM=1`), so it never engages here — which is priced into the next paragraph.
 
-## DSpark: an external drafter, measured on the 4-bit build
+**This build's own numbers, EOS-clean, same four-prompt protocol, greedy:** plain **21.8 tok/s** · MTP k=2 **31.1 (1.42×)** · DSpark at its 8-bit optimum, block 4, **33.6 (1.54×)**. The multiples are larger than the 4-bit build's because plain decode is slower here while drafting costs about the same — but mind the absolute ceiling: 8-bit with DSpark (33.6) is still slower than the 4-bit build running plain (37.6). Take this build for fidelity and speculate on it by all means; do not take it for speed.
 
-DSpark is the fork's other speculative path. Unlike MTP it is **not** self-speculation: it drives an external drafter, [`RadixArk/Qwen3.8-27B-DSpark`](https://huggingface.co/RadixArk/Qwen3.8-27B-DSpark) — 1.36B parameters, trained against this checkpoint, published by a third party. The drafter carries no embedding and no output head of its own; it borrows the target's, and drafts a whole block of positions in one forward pass instead of chaining them.
+## DSpark: an external drafter
+
+DSpark is the fork's other speculative path, and on this build it is the faster of the two — 33.6 tok/s (1.54×) at block 4, EOS-clean, against MTP k=2's 31.1. Unlike MTP it is **not** self-speculation: it drives an external drafter, [`RadixArk/Qwen3.8-27B-DSpark`](https://huggingface.co/RadixArk/Qwen3.8-27B-DSpark) — 1.36B parameters, trained against this checkpoint, published by a third party. The drafter carries no embedding and no output head of its own; it borrows the target's, and drafts a whole block of positions in one forward pass instead of chaining them.
 
 Two things to settle first:
 
@@ -194,23 +202,24 @@ prompt = mx.array(tokenizer.apply_chat_template(
     add_generation_prompt=True,
 ))
 
-for token, _accepted in dspark_generate_step(prompt, model, draft, max_tokens=512):
+for token, _accepted in dspark_generate_step(prompt, model, draft, max_tokens=512,
+                                             block_size=4):
     print(tokenizer.decode([token.item()]), end="", flush=True)
 ```
 
-The defaults are the measured operating point, not a permissive setting — block size 8, verification width capped at 8, the drafter's confidence head off — so the call above needs no tuning to reproduce the 4-bit row.
+**On this build, run block 4, not the block-8 default.** The default block size of 8 is the 4-bit operating point, where the small-M kernel makes wide verification windows nearly free. That kernel never engages at 8 bits, so width costs what it costs here and the measured optimum shrinks to block 4 — 36.3 tok/s against 35.1 at block 8 on the English-prompt block sweep, 33.6 on the full four-prompt set. The call above pins it. An earlier revision of this card warned that a pre-kernel 8-bit measurement ran DSpark below plain decode and that there might be "nothing here to collect" — the EOS-clean re-measurement retires that warning: 1.54× is real. Just weigh the download against the arithmetic above before collecting it.
 
-**Temper your expectations on this build specifically.** DSpark's margin over MTP rests on the kernel described above, which does not engage at 8 bits, and in pre-kernel measurements an 8-bit target ran DSpark **below** plain decode. It has not been re-measured since the kernel landed. Download 2.7 GB on the strength of the 4-bit row and you may find nothing here to collect.
+### Korean: the recommendation reversed
 
-### Do not turn speculative decoding on for Korean
+Until this revision the card said **"run Korean workloads plain,"** on the strength of 4-bit Korean cells that read below plain decode. **That was an artifact of the retracted protocol** — short answers plus a harness that measured past their end. Under the corrected protocol the gated MTP configuration on the 4-bit build is **faster on Korean than plain**: +34% greedy, +27–31% under the shipped sampling defaults. And on this build's own EOS-clean measurement both paths clear plain on Korean as well — MTP k=2 23.9 and DSpark 24.6 against 21.8 plain, greedy.
 
-The Korean column is the one to read before enabling any of this. On the Korean prompt **both** speculative paths came in **slower than plain decode** — DSpark 33.3 and MTP 34.3 against 37.6 tok/s plain — while the same two configurations returned 75.1 and 54.6 on the code prompt. Speculation only pays when the drafted tokens survive verification; on Korean they do not survive often enough to cover what drafting and verifying cost, so the machinery becomes pure overhead. **Run Korean workloads plain.**
+The honest exception sits on the 4-bit tier: **DSpark under the sampling defaults stays at or below plain on Korean there** (well below at 240 tokens, break-even at best by 1024). The reversal is about the gated MTP path and about this build; it is not a blanket license for every path on every tier.
 
 ### Measurement protocol, and why single-prompt numbers lie
 
-Every figure in the table is the **average over the fixed four-prompt set**, never a single run, and the spread inside one row is why. In the DSpark row the code prompt reads **75.1 tok/s** and the math prompt **91.5 tok/s**, while the honest four-prompt average is **62.2**. Quote the math prompt on its own and you are advertising 91.5 for a build that delivers 62.2 across mixed traffic — the same measurement, the same process, differing by nothing but the choice of prompt.
+Every figure above is the **average over the fixed four-prompt set**, never a single run. Speculative decoding pays off in proportion to how predictable the continuation is, so a benchmark built from one favorable prompt **overstates the gain you will see in real traffic** by a wide margin — treat any single-prompt speculative figure, including mine, as an upper bound.
 
-Speculative decoding pays off in proportion to how predictable the continuation is, so a benchmark built from one favorable prompt **overstates the gain you will see in real traffic** by a wide margin. Treat any single-prompt speculative figure, including mine, as an upper bound.
+The correction at the top of this section adds a second rule, learned at the cost of a retraction: **a speculative benchmark that does not stop at end-of-sequence is measuring luck** — what the model happens to do after its answer ends. The retracted 62.21 headline, and the 91.5 tok/s math cell behind it, were mostly that luck. The per-cell records behind the restated tables ship in the [campaign repository](https://github.com/avlp12/qwen38_alis_mlx).
 
 ## KV cache quantization
 
@@ -278,7 +287,7 @@ Two honest caveats on the table itself:
 - **Vision quality is unquantified.** The tower is preserved bit-exact and verified to work on a shapes probe. That is a functional check, not a VQA evaluation. No image benchmark was run at any bit-width.
 - **Shape-dependent numerics.** The hybrid stack takes different kernel paths depending on the shape of the input it receives — the batched scoring pass behind the perplexity table is not the same shape as incremental decode, and near-tie positions can resolve differently between those paths. Perplexity is far less sensitive to this than an argmax metric, but do not expect bit-identical numbers from a different harness.
 - **Long-form generated reasoning is unprobed.** Perplexity scores 2048-token windows of existing text; it does not exercise long generated reasoning chains, where low-bit damage tends to concentrate. Spot-check your own thinking-mode prompts.
-- **Speculative-decoding gains are workload-dependent, and the harness caveat applies.** See the footnotes under the speculative table.
+- **Speculative-decoding gains are workload-dependent.** The speculative tables were restated on 2026-08-16 under an EOS-cut protocol after the earlier harness was found to measure past end-of-sequence; the correction in the MTP section is the full account.
 
 ## License and attribution
 

assets/make_builds_png.py

@@ -17,7 +17,7 @@ import matplotlib
 
 matplotlib.use("Agg")
 import matplotlib.pyplot as plt
-from matplotlib.patches import Circle, FancyBboxPatch, Polygon
+from matplotlib.patches import FancyBboxPatch, Polygon
 
 plt.rcParams["font.family"] = ["Helvetica Neue", "Helvetica", "Arial", "DejaVu Sans"]
 
@@ -93,46 +93,28 @@ ax.plot([179.1, 275.0, 372.4, 736.1], [407.5, 478.9, 517.4, 581.8],
 line(100, 581.8, 800, 581.8, CB, 1.4, 0.8, dash=(6, 6), z=2)
 txt(108, 573.8, "bf16 baseline — 12.6 tok/s", 12.5, CB)
 
-# ============ speculative spread on the 4-bit build (all at 15.2 GB) ============
-# solid marker = 4-prompt average · caps = 3-English-prompt high and Korean low
-
-# upper spread: DSpark 4-prompt 62.21 -> 3-prompt 71.86
-line(179.1, 218, 179.1, 167, CD, 1.3, 0.55, z=3)
-line(171.1, 167, 187.1, 167, CD, 2.4, 1.0, z=4)
-txt(200, 171, "71.9 — 3 English prompts", 12.5, CD)
-
-# MTP 3-prompt cap 55.71
-line(172.1, 280, 186.1, 280, CM, 2.2, 1.0, z=4)
-txt(200, 284, "55.7 — 3 English prompts", 12.5, CM)
-
-# lower spread: plain 4-bit -> Korean-only lows
-line(179.1, 415.5, 179.1, 436.9, GREY, 1.3, 0.35, z=3)
-line(172.1, 429.9, 186.1, 429.9, CM, 2.2, 1.0, z=4)
-line(171.1, 436.9, 187.1, 436.9, CD, 2.4, 1.0, z=4)
-txt(166, 428, "ko 34.3", 12.5, CM, ha="right")
-txt(166, 446, "ko 33.3", 12.5, CD, ha="right")
-
-# ============ lift connectors: plain 37.6 -> MTP 50.36 -> DSpark 62.21 ============
-line(179.1, 399.5, 179.1, 329.5, CM, 1.6, 1.0, dash=(5, 5), z=3)
-txt(166, 370, "+34%", 14, CM, "bold", ha="right")
-
-line(179.1, 306.5, 179.1, 250.5, CD, 1.6, 1.0, dash=(5, 5), z=3)
-txt(166, 265, "+65%", 14, CD, "bold", ha="right")
-
-# ============ 4-bit + DSpark drafter (triangle, ringed) ============
-ax.add_patch(Circle((179.1, 234.5), 15.5, fill=False, edgecolor=CD,
-                    alpha=0.45, linewidth=1.3 * PT, zorder=5))
-ax.add_patch(Polygon([(179.1, 223.5), (191.1, 245.5), (167.1, 245.5)],
-                     closed=True, facecolor=CD, linewidth=0, zorder=6))
-txt(200, 229, "4-bit + DSpark drafter", 15.5, CD, "bold")
-txt(200, 248, "62.21 tok/s · 1.65× vs plain 4-bit", 13.5, CD)
-
-# ============ 4-bit + MTP k=2 (diamond) ============
-ax.add_patch(Polygon([(179.1, 306.5), (190.1, 317.5), (179.1, 328.5),
-                      (168.1, 317.5)],
+# ============ speculative ladder on the 4-bit build (all at 15.2 GB) ============
+# EOS-cut protocol, restated 2026-08-16 · greedy 4-prompt averages
+
+# lift connectors: plain 37.6 -> MTP k=2 46.8 -> gated MTP k=4 52.8
+line(179.1, 399.5, 179.1, 355.5, CM, 1.6, 1.0, dash=(5, 5), z=3)
+line(179.1, 330.4, 179.1, 312.4, CM, 1.6, 1.0, dash=(5, 5), z=3)
+
+# ============ 4-bit + gated MTP k=4 (filled diamond, top) ============
+ax.add_patch(Polygon([(179.1, 289.4), (190.1, 300.4), (179.1, 311.4),
+                      (168.1, 300.4)],
                      closed=True, facecolor=CM, linewidth=0, zorder=6))
-txt(200, 312, "4-bit + MTP k=2", 15.5, CM, "bold")
-txt(200, 331, "50.36 tok/s · 1.34× vs plain 4-bit", 13.5, CM)
+txt(166, 304, "+40%", 14, CM, "bold", ha="right")
+txt(200, 296, "4-bit + MTP k=4, p-gated", 15.5, CM, "bold")
+txt(200, 315, "52.8 tok/s · 1.40× vs plain 4-bit (greedy)", 13.5, CM)
+
+# ============ 4-bit + MTP k=2 (open diamond) ============
+ax.add_patch(Polygon([(179.1, 331.4), (190.1, 342.4), (179.1, 353.4),
+                      (168.1, 342.4)],
+                     closed=True, fill=False, edgecolor=CM,
+                     linewidth=2 * PT, zorder=6))
+txt(166, 346, "+24%", 14, CM, "bold", ha="right")
+txt(200, 347, "MTP k=2 — 46.8 tok/s · 1.24×", 13.5, CM)
 
 # ============ 4-bit ============
 ax.plot([179.1], [407.5], "o", markersize=16 * PT, color=C4, zorder=6)
@@ -158,14 +140,15 @@ txt(790, 512, "bf16 reference", 15.5, CB, "bold", ha="right")
 txt(790, 531, "51.8 GB · 12.6 tok/s", 13.5, CB, ha="right")
 txt(790, 549, "ko PPL 6.095 — reference", 12.5, CB, ha="right")
 
-# ============ in-plot callout: the Korean caveat ============
+# ============ in-plot callout: the restatement ============
 ax.add_patch(FancyBboxPatch((459.5, 190.5), 0.1, 15,
                             boxstyle="round,pad=1.5,rounding_size=1.5",
                             linewidth=0, facecolor=CD, zorder=4))
-txt(470, 200, "Speculation is prompt-dependent", 15, INK, "bold")
-txt(470, 222, "Plotted values are 4-prompt averages. On the three", 13, MUT)
-txt(470, 240, "English prompts DSpark reads 71.9 tok/s; on Korean", 13, MUT)
-txt(470, 260, "alone 33.3 — below plain 37.6. Run Korean plain.", 13, MUT)
+txt(470, 200, "Restated 2026-08-16 (EOS-cut)", 15, INK, "bold")
+txt(470, 222, "Speculative points are EOS-cut 4-prompt averages,", 13, MUT)
+txt(470, 240, "greedy. The old 62.2 / 50.4 headline decoded past", 13, MUT)
+txt(470, 258, "EOS and is retracted. DSpark restates to 48.3", 13, MUT)
+txt(470, 276, "(1.28×); Korean gains +27–31% with the gated path.", 13, MUT)
 
 # ============ side facts ============
 ax.add_patch(FancyBboxPatch((836, 122), 330, 468,
@@ -194,7 +177,8 @@ txt(858, 438, "with --kv-bits 4 — top-1 unchanged.", 13, MUT)
 txt(858, 498, "Speculation is opt-in", 15, INK, "bold")
 txt(858, 520, "Both paths need the avlp12/mlx-lm fork.", 13, MUT)
 txt(858, 538, "The MTP head ships inside this build; the", 13, MUT)
-txt(858, 556, "DSpark drafter is a separate 0.76 GB download.", 13, MUT)
+txt(858, 556, "DSpark drafter is a separate 0.76 GB", 13, MUT)
+txt(858, 574, "download — greedy 48.3 tok/s (1.28×).", 13, MUT)
 
 # ============ footnotes ============
 for fy, s in [
@@ -204,15 +188,16 @@ for fy, s in [
     (701, "wiki 35,179, code CPython 32,295 tokens. 8-bit ≈ bf16 on every "
           "slice, 6-bit ≈ bf16 on Korean; 4-bit AWQ: +1.2% en, +2.7% ko, "
           "+7.7% code. No standard benchmarks were run."),
-    (720, "Speculative decoding: greedy, 240 tokens per prompt, timed after an "
-          "8-token warmup, in one process on the 4-bit repo's earlier uniform "
-          "weights (the AWQ swap measures within ±1%,"),
-    (739, "EOS-cut paired). Both lossless — each reproduced plain greedy "
-          "decoding token for token through 80 tokens. MTP is the vendor "
-          "multi-token head at k=2; DSpark drives an external 1.36B drafter."),
-    (758, "Plotted values are the 4-prompt average — single prompts overstate "
-          "speculation: DSpark reads 75.1 tok/s on code and 91.5 on math "
-          "against that 62.2 average.")]:
+    (720, "Speculative decoding (restated 2026-08-16): EOS-cut, long-form four "
+          "prompts (chat/code/math/Korean), medians of 3 for sampled rows, one "
+          "process, on the 4-bit repo's earlier uniform weights"),
+    (739, "(the AWQ swap measures within ±1%, paired). Greedy: plain 37.6 · "
+          "MTP k=2 46.8 (1.24×) · DSpark block-8 48.3 (1.28×) · MTP k=4 + "
+          "min_draft_p 0.6 = 52.8 (1.40×). The old 62.2/50.4 figures"),
+    (758, "decoded past EOS; retracted. Real-world sampling (temp 1.0 · top-p "
+          "0.95 · top-k 20): gated MTP 1.29× @240 tok, 1.22× @1024, "
+          "distribution-preserving; Korean +27–31%; plain pays −1.2% for the "
+          "sampler.")]:
     txt(40, fy, s, 12, MUT)
 
 out = "/Users/gesicht/qwen38/cards/assets/builds.png"