SPEED.md

# Speed — the MEASURED reality (M5 Max 128 GB)

**Root cause (confirmed): decode is memory-bandwidth bound** — every token reloads expert weights from the
~106 GB model. The honest conclusion that follows was *measured*, not guessed (2026-06-18):

> **Every single-stream speedup lever has been benchmarked, and every one is dead on this MoE.**
> ~11–14 tok/s is the hard memory floor. You cannot speculate your way under it.

| Lever | Hoped | **MEASURED** | Why it fails |
|---|---|---|---|
| MTP self-speculative | ~2.6× | **0% accept** (`89_mtp_gate.py`) | the native head is UN-pruned (256 experts) vs our pruned 77 → its MoE/router don't match → garbage drafts |
| External draft model | 1.5–2.5× | **0.32× (proxy)** | the batched *verify* forward reloads ~all 77 experts; also draft+main ≈126/128 GB + Metal-unstable |
| Prompt-lookup (no model) | 2–4× | **0.32×** (`src/prompt_lookup.py`) | lossless ✓ but the same MoE verify-wall — 8.4 tok/forward, but each forward loads ~all experts |
| dsa-block-size / index_topk | ~1.3× | **flat 1.00–1.03×** (`90_dsa_sweep.py`) | attention is NOT the bottleneck; the MoE expert-load dominates |
| Reduce active experts 8→6 | ~25% | **quality-dead** (`22_speed_tune.py`) | the router was trained for top-8 of the pruned 77; can't drop post-prune without a full re-prune+heal |

### The physics
A MoE decode step is bound by loading the active experts' weights. Any *multi-token* speculative forward
(MTP, draft-model, or n-gram) loads the **union** of experts for those tokens (~all 77) → costs ~K× a single
step → the per-forward token gain is exactly cancelled. So **no speculative method beats single-token decode here.**
Proven 4 independent ways, $0 spent (vs the $4–15K an EAGLE retrain would have cost to discover the same thing).

### What ACTUALLY delivers speed (and is already shipped)
1. **Fused MoE dequant-matmul Metal kernel** (#33) — this *is* the 11–14 tok/s; it's the real win.
2. **Throughput via batching — MEASURED 2.6× at B=8** (`scripts/91_batch_scaling.py`): total decode
   **15.8 → 27.1 → 34.6 → 41.1 tok/s** at B = 1 / 2 / 4 / 8. On a memory-bound MoE, *parallel sequences* are
   the lever — one expert-load serves the whole batch (per-seq drops, total climbs). This is where "faster"
   actually lives (best-of-N, proof-search, multi-agent, the flywheel), and `mlx_lm.server` batches concurrent
   requests **natively** (`is_batchable`, no draft model — that's us), so the win is available at the serve (#71).
   **Serve-verified: 1.74× at B=6** on the live `mlx_lm.server` (concurrent vs sequential, `scripts/92_serve_batch_test.py`)
   — already shipping; just fire concurrent requests. The ONLY measured speedup on this model that beats 1×.

### Free, orthogonal wins (prefill / TTFT — not decode tok/s)
- **Prompt/prefix KV cache** (`05_serve.sh --prompt-cache-size`) — agentic loops resend the same system+tools;
  cache the prefix → big *time-to-first-token* cut. Real, but it doesn't move the decode tok/s number.
- **M5 Neural Accelerators / current MLX** — free generation + prefill gains from keeping `mlx` current.
- **`08_think_proxy.py`** — skip the thinking trace on trivial structural steps → fewer tokens, not faster/token.

### The ONLY path to a single-stream speedup (not recommended)
Train a **fresh EAGLE-3 head** on the demolished model's own outputs (architecture-agnostic, ~1B dense layer):
~$4–15K cloud H100 or ~weeks of local M5 data-gen. And even then it's **uncertain** — EAGLE on a fast MoE
baseline measured ~1.03× in the literature (same verify-wall). See `#69`. Bank the batching instead.

### Bottom line
Single-stream is **memory-capped at ~11–14 tok/s and that's the floor**. The speed pillar is the fused kernel
(#33) + throughput batching (#35/#48), both built. Don't fight physics; batch for throughput.

*(Receipts: `89_mtp_gate.py` → 0%, `src/prompt_lookup.py --bench` → 0.32×, `90_dsa_sweep.py` → flat. Full log
in OVERNIGHT_LOG.md under the 2026-06-18 speed entries.)*