Quality: corpus strided PPL replaces low-power top-1 probe; spec-table harness footnotes; chart updated

README.md

@@ -17,20 +17,20 @@ tags:
 
 # Qwen3.8-27B-Alis-MLX-8bit
 
-![Build size vs decode speed across the 8-bit, 6-bit and 4-bit builds, with Korean top-1 agreement and both speculative paths, MTP k=2 and the DSpark drafter](assets/builds.png)
+![Build size vs decode speed across the 8-bit, 6-bit and 4-bit builds, with Korean corpus perplexity and both speculative paths, MTP k=2 and the DSpark drafter](assets/builds.png)
 
-An 8-bit MLX quantization of [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) for Apple silicon, with the **vision tower and the MTP head both preserved**. 27.9 GB on disk, 21.8 tok/s decode, and top-1 agreement with the bf16 original of 99.1 / 98.4 / 100 on Korean / English / code. This is the fidelity build of the set.
+An 8-bit MLX quantization of [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) for Apple silicon, with the **vision tower and the MTP head both preserved**. 27.9 GB on disk, 21.8 tok/s decode, and corpus perplexity **statistically indistinguishable from the bf16 original** on English, Korean, and code alike — paired over ≈103K scored tokens. This is the fidelity build of the set.
 
 ## At a glance
 
-| Build | Size (GB) | Decode (tok/s) | Prefill (tok/s) | Peak RAM (GB) | Korean NLL | top-1 ko / en / code |
-|---|---|---|---|---|---|---|
-| bf16 reference | 51.8 | 12.6 | 410 | 51.1 | 1.661 | 100 / 100 / 100 |
-| 8-bit (this repo) | 27.9 | 21.8 | 429 | 28.2 | 1.663 | 99.1 / 98.4 / 100 |
-| 6-bit | 21.5 | 27.3 | 424 | 21.9 | 1.679 | 97.3 / 96.8 / 100 |
-| 4-bit | 15.2 | 37.5 | 436 | 15.6 | 1.679 | 85.7 / 93.5 / 100 |
+| Build | Size (GB) | Decode (tok/s) | Prefill (tok/s) | Peak RAM (GB) | PPL en / ko / code |
+|---|---|---|---|---|---|
+| bf16 reference | 51.8 | 12.6 | 410 | 51.1 | 5.7734 / 6.0954 / 1.6813 |
+| 8-bit (this repo) | 27.9 | 21.8 | 429 | 28.2 | 5.7760 / 6.0987 / 1.6815 |
+| 6-bit | 21.5 | 27.3 | 424 | 21.9 | 5.7924 / 6.1018 / 1.6854 |
+| 4-bit (AWQ) | 15.2 | 37.5 | 436 | 15.6 | 5.8450 / 6.2609 / 1.8105 |
 
-Measured on one Apple M3 Ultra (512 GB unified memory), all builds queued back to back in a single batch so they share machine state. Peak RAM is the load-plus-probe peak at short context; add KV cache for your own context length (see below).
+Speed and RAM were measured on one Apple M3 Ultra (512 GB unified memory), all builds queued back to back in a single batch so they share machine state; the 4-bit row of that batch was that repo's earlier uniform release, and its AWQ replacement is the same size with plain decode re-measured at 37.7 tok/s. Peak RAM is the load-plus-probe peak at short context; add KV cache for your own context length (see below). Perplexity is corpus-scale strided PPL — ctx 2048, stride 512 — described in the quality section.
 
 Against bf16 this build is 54% of the size at 1.73× the decode speed, and it is the closest of the three to the original's token-level behavior.
 
@@ -74,13 +74,13 @@ python -m mlx_vlm.generate \
   --prompt "Describe every shape in this image, with its color and position."
 ```
 
-Verified end to end before publishing: I fed a hand-drawn test image (red circle top left, blue square top right, green triangle at the bottom) to the 4-bit build of this set and it described every color, shape, and position correctly. The vision path is the same bf16 tower in all three builds.
+Verified end to end before publishing: I fed a hand-drawn test image (red circle top left, blue square top right, green triangle at the bottom) to the 4-bit build of this set — its earlier uniform release — and it described every color, shape, and position correctly. The vision path is the same bf16 tower in all three builds.
 
 ## Which build should I use?
 
-- **8-bit (this repo, 27.9 GB)** — the fidelity choice. Take it when the output is the product: Korean or other non-English generation, long-form writing, agent loops where one diverged token compounds, or any run you intend to use as the reference when comparing the smaller builds. Its Korean NLL of 1.663 sits 0.002 above bf16, which is as close as this set gets. Plan for a 48 GB machine or larger: peak was 28.2 GB before any meaningful context, and 34.46 GB at 16K.
-- **6-bit (21.5 GB)** — the balanced default. 23% smaller and 25% faster than this build, holding Korean top-1 agreement at 97.3. Comfortable on a 32 GB Mac.
-- **4-bit (15.2 GB)** — the reach build, and the fastest at 37.5 tok/s. Choose it when the model would otherwise not fit, or for code and English work where the loss is smallest. Its Korean top-1 agreement falls to 85.7; do not put it in front of Korean-speaking users without your own evaluation.
+- **8-bit (this repo, 27.9 GB)** — the fidelity choice. Take it when the output is the product: Korean or other non-English generation, long-form writing, agent loops where one diverged token compounds, or any run you intend to use as the reference when comparing the smaller builds. On corpus perplexity a paired test over ≈103K tokens cannot separate it from bf16 on any slice — as close as this set gets. Plan for a 48 GB machine or larger: peak was 28.2 GB before any meaningful context, and 34.46 GB at 16K.
+- **6-bit (21.5 GB)** — the balanced default. 23% smaller and 25% faster than this build, with Korean corpus PPL still statistically indistinguishable from bf16 and only hairline (though significant) excesses on English and code. Comfortable on a 32 GB Mac.
+- **4-bit (15.2 GB, now the AWQ recipe)** — the reach build, and the fastest at 37.5 tok/s. Choose it when the model would otherwise not fit. It is the one build whose quality cost is significant on every slice — corpus PPL +1.2% / +2.7% / +7.7% over bf16 on English / Korean / code; do not put it in front of Korean-speaking users without your own evaluation.
 
 All three carry the same vision tower and the same MTP head. The choice is purely language-model precision.
 
@@ -148,7 +148,7 @@ mlx_lm.generate --model avlp12/Qwen3.8-27B-Alis-MLX-8bit \
   --max-tokens 512
 ```
 
-**Every speculative figure below was measured on the 4-bit build of this set, not on this one.** It is quoted here because it is what the fork's two speculative paths do on this checkpoint, and because the ordering between them is informative — but the multiples do not transfer to this build, for a reason spelled out under the table. Measured on one Apple M3 Ultra (512 GB), greedy, 240 tokens per prompt, timed after an 8-token warmup, every configuration in the same process, over four prompts: chat, code, math, and Korean.
+**Every speculative figure below was measured on the 4-bit build of this set — its earlier uniform 4-bit weights — not on this one.** It is quoted here because it is what the fork's two speculative paths do on this checkpoint, and because the ordering between them is informative — but the multiples do not transfer to this build, for a reason spelled out under the table. Measured on one Apple M3 Ultra (512 GB), greedy, 240 tokens per prompt, timed after an 8-token warmup, every configuration in the same process, over four prompts: chat, code, math, and Korean.
 
 | Configuration, 4-bit build | 4-prompt avg | 3 English prompts | chat | code | math | ko |
 |---|---|---|---|---|---|---|
@@ -158,6 +158,8 @@ mlx_lm.generate --model avlp12/Qwen3.8-27B-Alis-MLX-8bit \
 
 Lossless on that build: all four prompts reproduced plain greedy decoding **token for token through 80 tokens**.
 
+**Three footnotes on this table, added after an audit of the measurement harness.** First, the harness did **not stop at end-of-sequence** — every run decoded a fixed 240 tokens, so on prompts whose answer ends early, part of the window measures post-termination behavior, and per-prompt figures are sensitive to what a build happens to do after its answer ends. Read the averages, not the per-prompt cells. Second, acceptance-rate comparisons **between builds** are only meaningful on a common token sequence, scored paired — each build free-running its own text measures the text as much as the build. Third, the 4-bit build measured here was that repo's earlier uniform release; the AWQ weights now hosted there measure **within ±1%** of it on both speculative paths under an EOS-cut paired protocol. A full re-measurement under the corrected harness is a separate track.
+
 **Why this build cannot inherit those multiples.** What lifted them is a small-M quantized-GEMM kernel that the fork now enables from inside `mlx_lm.utils.load()`. It fires only in the multi-token verification window (M = 6..8) and only on **4-bit, group-64** weights, so it never engages on an 8-bit build; plain decode at M = 1 is untouched on every build either way. The gap it fills is upstream: [ml-explore/mlx#4265](https://github.com/ml-explore/mlx/issues/4265). Set `MLXLM_NO_FAST_QMM=1` to opt out.
 
 On this build MTP was measured at **21.9 to 31.3 tok/s, a 43% gain** — but treat that as a **pre-kernel measurement on an older three-prompt English-and-code set**, not a row of the table above, and not re-measured since. k=2 remains the sensible starting point here; measure your own workload rather than borrowing a number from either source.
@@ -212,7 +214,7 @@ Speculative decoding pays off in proportion to how predictable the continuation
 
 The hybrid layout is why long context is cheap on this model. Only the **16 full-attention layers** hold a growing KV cache; the 48 GatedDeltaNet layers carry a fixed-size recurrent state instead. With 4 KV heads at head_dim 256, that comes to **64 KB per token** in bf16 across the whole model.
 
-Measured at 16K context, as whole-process peaks — so they include activations and prefill buffers, not the cache alone:
+Measured at 16K context (the 4-bit column on that repo's earlier uniform release), as whole-process peaks — so they include activations and prefill buffers, not the cache alone:
 
 | KV precision | Peak RAM (GB), this build | Peak RAM (GB), 4-bit build | top-1 agreement |
 |---|---|---|---|
@@ -249,32 +251,32 @@ This is a flat, data-free round-to-nearest recipe: no calibration set, no activa
 
 ## Quality
 
-Teacher-forced against the bf16 original on three held-out passages — Korean prose (112 tokens), English prose (62 tokens), and Python (91 tokens) — in a single pass, reporting **top-1 agreement**: the fraction of positions where this build's argmax matches bf16's argmax on identical context. Korean NLL is the mean negative log-likelihood on that same Korean passage.
+Corpus-scale strided perplexity against fixed token streams: window 2048, stride 512, so every scored token sees up to 2048 tokens of left context. Three slices — **English** = wikitext-2 test (35,981 tokens), **Korean** = Korean Wikipedia (35,179 tokens), **code** = CPython standard library (32,295 tokens). Every build scores the same tokens in the same order, so build-to-build comparisons are paired per token, and significance is judged on the paired excess NLL with a blockwise standard error (512-token blocks) — roughly 16–20× the sensitivity of the probe this section used to carry.
 
-| Slice | bf16 | 8-bit (this repo) | 6-bit | 4-bit |
-|---|---|---|---|---|
-| Korean top-1 | 100 | 99.1 | 97.3 | 85.7 |
-| English top-1 | 100 | 98.4 | 96.8 | 93.5 |
-| Code top-1 | 100 | 100 | 100 | 100 |
-| Korean NLL | 1.661 | 1.663 | 1.679 | 1.679 |
+| Build | en · wikitext-2 | ko · Korean wiki | code · CPython |
+|---|---|---|---|
+| bf16 reference | 5.7734 | 6.0954 | 1.6813 |
+| **8-bit (this repo)** | **5.7760** | **6.0987** | **1.6815** |
+| 6-bit | 5.7924 | 6.1018 | 1.6854 |
+| 4-bit (AWQ) | 5.8450 | 6.2609 | 1.8105 |
 
-This is the build that holds. Korean NLL sits 0.002 above bf16 and fewer than one Korean token in a hundred takes a different top-1. If you want a local reference for judging the smaller builds, this is it.
+This is the build that holds — and now with a test strong enough to mean it: **on every slice the paired 95% confidence interval on this build's excess NLL over bf16 includes zero.** The corpus test that comfortably separates the 4-bit build (+1.2% / +2.7% / +7.7% PPL on en / ko / code) and even the 6-bit build's hairline excesses on English and code cannot tell this build from the original. If you want a local reference for judging the smaller builds, this is it.
 
 Two honest caveats on the table itself:
 
-- **Code being flat at 100 across every build does not prove 8-bit is lossless.** Code is the most predictable of the three slices; it failed to separate any build and therefore carries no evidence about them.
-- **top-1 agreement is a strict positional metric, not a score.** A mismatch is not automatically a worse token, and it does not translate into a percentage of wrong output. It is a sensitive damage detector, used here to locate damage rather than to price it.
+- **"Indistinguishable" means the test could not separate them at this sample size, not that they are identical.** A still larger corpus might resolve a difference; ≈103K paired tokens did not.
+- **Perplexity prices average next-token surprise, not task success.** It ranks the builds with real statistical power, but it is not a benchmark suite.
 
-Note also that the 6-bit and 4-bit builds record an identical Korean NLL of 1.679 while their top-1 agreement differs by 11.6 points. Two readings fit that: the 4-bit damage is concentrated at near-tie positions where the ranking flips but the probability mass barely moves, or a 112-token passage is simply too short for NLL to separate them. I did not run further probes to decide between the two, so treat both rows as coarse instruments.
+**Why this table changed.** Earlier revisions of this card ranked the builds with a top-1 agreement probe over three short passages (62–112 tokens). At that size the probe had no statistical power: its standard error ran several times the effects it claimed to rank, and on English it read the sign of a recipe difference **backwards**. It is removed rather than restated.
 
 ## Limitations and caveats
 
-- **Probe-scale measurement, not a benchmark suite.** Quality comes from a single teacher-forced pass over three short passages (112 / 62 / 91 tokens), greedy argmax.
-- **No standard benchmarks were run.** There are no MMLU, GSM8K, or HumanEval numbers here, and none should be inferred from the agreement table.
+- **Perplexity is the only quality metric here.** It is corpus-scale (≈103K scored tokens, paired per token) and it replaced a probe that had no statistical power — but it is still average next-token surprise, not a task suite.
+- **No standard benchmarks were run.** There are no MMLU, GSM8K, or HumanEval numbers here, and none should be inferred from the perplexity table.
 - **Vision quality is unquantified.** The tower is preserved bit-exact and verified to work on a shapes probe. That is a functional check, not a VQA evaluation. No image benchmark was run at any bit-width.
-- **Shape-dependent numerics.** The hybrid stack takes different kernel paths depending on the shape of the input it receives — a batched teacher-forced pass is not the same shape as incremental decode. On the same prefix, positions whose top-2 logits are nearly tied can flip between those paths. Agreement figures carry a run-to-run wobble at this precision; do not read the last decimal as stable.
-- **Long-form reasoning is unprobed.** Every probe here is short, while low-bit damage tends to concentrate in long reasoning chains. Spot-check your own thinking-mode prompts.
-- **Speculative-decoding gains are workload-dependent.** See the measurement protocol above.
+- **Shape-dependent numerics.** The hybrid stack takes different kernel paths depending on the shape of the input it receives — the batched scoring pass behind the perplexity table is not the same shape as incremental decode, and near-tie positions can resolve differently between those paths. Perplexity is far less sensitive to this than an argmax metric, but do not expect bit-identical numbers from a different harness.
+- **Long-form generated reasoning is unprobed.** Perplexity scores 2048-token windows of existing text; it does not exercise long generated reasoning chains, where low-bit damage tends to concentrate. Spot-check your own thinking-mode prompts.
+- **Speculative-decoding gains are workload-dependent, and the harness caveat applies.** See the footnotes under the speculative table.
 
 ## License and attribution
 

assets/make_builds_png.py

@@ -136,35 +136,27 @@ txt(200, 331, "50.36 tok/s · 1.34× vs plain 4-bit", 13.5, CM)
 
 # ============ 4-bit ============
 ax.plot([179.1], [407.5], "o", markersize=16 * PT, color=C4, zorder=6)
-txt(200, 366, "4-bit", 15.5, C4, "bold")
+txt(200, 366, "4-bit AWQ", 15.5, C4, "bold")
 txt(200, 385, "15.2 GB · 37.5 tok/s plain", 13.5, C4)
-txt(200, 403, "ko top-1 85.7%", 12.5, C4, "bold")
-bar(200, 411, 84, GREY, 0.16, z=4)
-bar(200, 411, 23.9, C4)
+txt(200, 403, "ko PPL 6.261 · +2.7% vs bf16", 12.5, C4, "bold")
 
 # ============ 6-bit (label to the left) ============
 ax.plot([275.0], [478.9], "o", markersize=16 * PT, color=C6, zorder=6)
 txt(257, 482, "6-bit", 15.5, C6, "bold", ha="right")
 txt(257, 501, "21.5 GB · 27.3 tok/s", 13.5, C6, ha="right")
-txt(257, 519, "ko top-1 97.3%", 12.5, C6, ha="right")
-bar(173, 527, 84, GREY, 0.16, z=4)
-bar(173, 527, 72.7, C6)
+txt(257, 519, "ko PPL 6.102 (≈bf16)", 12.5, C6, ha="right")
 
 # ============ 8-bit ============
 ax.plot([372.4], [517.4], "o", markersize=16 * PT, color=C8, zorder=6)
 txt(390, 459, "8-bit", 15.5, C8, "bold")
 txt(390, 478, "27.9 GB · 21.8 tok/s", 13.5, C8)
-txt(390, 496, "ko top-1 99.1%", 12.5, C8)
-bar(390, 504, 84, GREY, 0.16, z=4)
-bar(390, 504, 80.2, C8)
+txt(390, 496, "ko PPL 6.099 (≈bf16)", 12.5, C8)
 
 # ============ bf16 ============
 ax.plot([736.1], [581.8], "o", markersize=16 * PT, color=CB, zorder=6)
 txt(790, 512, "bf16 reference", 15.5, CB, "bold", ha="right")
 txt(790, 531, "51.8 GB · 12.6 tok/s", 13.5, CB, ha="right")
-txt(790, 549, "ko top-1 100%", 12.5, CB, ha="right")
-bar(706, 557, 84, GREY, 0.16, z=4)
-bar(706, 557, 84, CB)
+txt(790, 549, "ko PPL 6.095 — reference", 12.5, CB, ha="right")
 
 # ============ in-plot callout: the Korean caveat ============
 ax.add_patch(FancyBboxPatch((459.5, 190.5), 0.1, 15,
@@ -206,18 +198,18 @@ txt(858, 556, "DSpark drafter is a separate 0.76 GB download.", 13, MUT)
 
 # ============ footnotes ============
 for fy, s in [
-    (682, "Measured on one M3 Ultra 512GB box, queued back-to-back. Bars show "
-          "Korean top-1 agreement vs bf16 on an 80–100% scale; the probe is a "
-          "teacher-forced single pass over"),
-    (701, "three passages (ko 112 / en 62 / code 91 tokens). At 4-bit, English "
-          "is 93.5% and code 100%, and Korean is 11.6 points behind 6-bit. "
-          "No standard benchmarks were run."),
+    (682, "Measured on one M3 Ultra 512GB box, queued back-to-back. Quality = "
+          "corpus strided perplexity — ctx 2048, stride 512, paired per token "
+          "vs bf16 — over en wikitext-2 35,981, ko Korean"),
+    (701, "wiki 35,179, code CPython 32,295 tokens. 8-bit ≈ bf16 on every "
+          "slice, 6-bit ≈ bf16 on Korean; 4-bit AWQ: +1.2% en, +2.7% ko, "
+          "+7.7% code. No standard benchmarks were run."),
     (720, "Speculative decoding: greedy, 240 tokens per prompt, timed after an "
-          "8-token warmup, every configuration in one process on this 4-bit "
-          "build. Both paths are lossless — through"),
-    (739, "80 tokens each reproduced plain greedy decoding token for token. "
-          "MTP is the checkpoint's own vendor multi-token-prediction head at "
-          "k=2; DSpark drives an external 1.36B drafter."),
+          "8-token warmup, in one process on the 4-bit repo's earlier uniform "
+          "weights (the AWQ swap measures within ±1%,"),
+    (739, "EOS-cut paired). Both lossless — each reproduced plain greedy "
+          "decoding token for token through 80 tokens. MTP is the vendor "
+          "multi-token head at k=2; DSpark drives an external 1.36B drafter."),
     (758, "Plotted values are the 4-prompt average — single prompts overstate "
           "speculation: DSpark reads 75.1 tok/s on code and 91.5 on math "
           "against that 62.2 average.")]: