# Roadmap — open work only **This is the only list of open work.** Anything finished moves to [ARCHIVE.md](ARCHIVE.md) with the reasoning that produced it. [AI.md](AI.md) is the reference for what is true and measured; [API.md](API.md) is the full call surface, asserted against the code; [WEBLLM-SURFACE.md](WEBLLM-SURFACE.md) is what WebLLM already does and must be read before adding a capability. Sections are named, not numbered. An earlier split across three files used "Track 1"/"Track 2" to mean different things in each, which is how a reader ends up implementing the wrong item. **Sequencing.** The whole project exists to make WebLLM *more compatible, easier to use, foolproof to build on*. Every item below is ranked against that. `everything-webgpu@0.1.0` is on npm as of this writing — the install path, the four-line demo, the ergonomic verbs, `API.md` and `examples/` all shipped and are recorded in [ARCHIVE.md](ARCHIVE.md). Section 1 is what remains before the extraction is fully proven: the extension itself rebuilding on the package. Section 4 (performance) is frozen as a block, except for "Measure Ollama", until a real user calls it slow: it is the highest-risk, highest-effort work and it re-touches the tvmjs internals the WebLLM-upgrade section (also in [ARCHIVE.md](ARCHIVE.md)) just spent its time hardening. --- ## 1. Ship it — the last mile `everything-webgpu@0.1.0` is on npm. What shipped — the install path, the four-line demo, the ergonomic verbs, [API.md](API.md), [examples/](examples/), the bundle-size story, licence compliance — is in [ARCHIVE.md](ARCHIVE.md), "Shipping 0.1.0". What remains is proving the boundary from the other side. - [ ] **`demo` consumes the package**, and the popup + manager leave `main`. The acceptance test for the whole extraction: if the extension rebuilds on the published package, the boundary is right. `npm run e2e` passing is the same claim at the source-tree level. - [~] **Gate A — a bare Vite page loads a prebuilt model and generates.** [examples/bare/](examples/bare/) exists and the GPU-free half is proven: `vite build` resolves through the `exports` map and splits into an entry chunk, the decode worker as its own chunk, and the 6 MB WebLLM bundle as a lazy one. What remains is **one run in a WebGPU browser** to confirm it generates. `npm run verify-consumer` already exercises a real install end to end, short of the actual generation. - [ ] **Defect: Worker unbundled relative imports in production (`vite build` / Rollup blind spot).** First real-browser run on a static host (Hugging Face Spaces) surfaced a packaging defect that passed all prior headless assertions. On `vite build`, `engine.load()` threw `PACKAGE_INCOMPLETE`: *"The decode worker failed to load from .../assets/engine-worker-*.js"*. **Root cause:** Rollup's `new Worker(new URL(..., import.meta.url), { type: "module" })` handling copies `engine-worker.js` as an asset chunk without bundling or rewriting its internal ESM imports. The emitted worker still carries: `import { WebWorkerMLCEngineHandler } from "../../vendor/web-llm.js"` `import { WORKER_CONFIGURE } from "./constants.js"` `import { DEFAULT_DECODE_STEPS } from "./multistep.js"` On a static production deployment, these relative URLs resolve to 404s on the root domain, killing the worker silently before handshake. `vite dev` hid this because `optimizeDeps.exclude` served files directly from source where siblings exist. `test/consumer/verify.mjs` line 130 missed it because it asserted file *existence* (`assets.some(f => f.startsWith("engine-worker"))`), never executing the worker in a browser. **Resolution:** - [x] **Demo workaround:** Pre-bundle `engine-worker.js` with `esbuild --bundle --format=esm` into a standalone 6.58 MB file (`public/engine-worker.js`) with zero external imports, and pass explicit `workerUrl: new URL("engine-worker.js", window.location.href)`. - [ ] **Library permanent fix for 0.1.1:** In `build.mjs`, pre-bundle `engine-worker.js` into a self-contained `dist/worker.bundle.js` at package build time. Point `DEFAULT_WORKER_URL` at the bundle so zero consumer configuration or `workerUrl` is required. Update `verify-consumer` with a headless browser fetch/execute assertion to prevent regressions. - [ ] **Gate A′ — the same page ingests a local folder.** `cache.put()` against the synthetic `local-model.invalid` key is proven only on an extension origin. Gates the *offline* route, not the library. - [ ] **Gate B — Chrome.** Measure tok/s. Expected to beat Firefox because KV reuse is not disabled there. Entirely a prediction today; `probeDevice()` is the instrument that makes it reportable. - [ ] **Every `ERROR` code carries an actionable `fix`.** The preflight (`environment()`) and the typed errors (9 codes now, `PACKAGE_INCOMPLETE` included) both exist; what is not yet done is the systematic pass confirming each code's `message`/`detail` names a cause *and* a fix, and that the README/API.md point at `environment()` as the "will this run here?" call it is meant to be. ## 2. Verb consolidation — done `load(src, opts)`, `unload(id, level)` / `remove()` / `unloadAll()`, and `environment()` + `environment.measure()` all shipped. The reasoning — the two dropped `load()` dispatch rules, why a bare `unload()` frees only the current model, why read and write are split — is in [ARCHIVE.md](ARCHIVE.md), "Verb consolidation". `chat.completions.create()` is unchanged and stays that way. ## 3. Engine capability `prefetch()`, `embed()` / `embedRaw()`, and the `ask()` / `conversation()` / `ghostText()` recipes shipped — see [ARCHIVE.md](ARCHIVE.md), "Engine capability". One item is still open: - [ ] **LRU eviction** when quota is short. `cacheState()` and `evict()` exist; nothing yet decides *what* to drop. Table stakes for "it just works" the moment a second model is cached — but the eviction policy is a decision, not an implementation detail. ## 4. Performance — frozen, except the measurement Model- and kernel-level work, independent of the library structure — and therefore independent of everything this project is *for*. **Frozen as a block until a real user calls it slow.** The library ships now, so "nobody can install it" is no longer the argument; the argument is that this is the highest-risk, highest-effort work on the page, it re-touches the tvmjs internals the WebLLM-upgrade section (in [ARCHIVE.md](ARCHIVE.md)) just spent its time hardening, and a speedup nobody has asked for is risk without a return. The one exception is "Measure Ollama": it is cheap, and it decides whether any of the rest is worth its risk. - [ ] **Retune the dlight GEMV schedule** — more work per thread before the reduction. Measured **1.83x** in isolation, which would put decode near the ~41 GB/s dequant ceiling. Confirmed untaken on the 2B build. Needs a recompile; toolchain is stood up. - [ ] **Batched decode.** The model lib exports `batch_decode` / `batch_prefill` / `batch_verify` and a paged KV cache; WebLLM hardcodes `defaultMaxNumSequence = 1` and `numSamples = 1` (bundle lines 15250, 15276). Lifting that reads the weights once per step for N sequences — projected **~4x on the `batch` API**, which is the page-translation shape and the one workload the engine currently makes no faster than a loop of `chat` calls. No recompilation needed. **Not** a fix for the 1.06x second-engine measurement: a second engine buys task isolation, never aggregate throughput, and the two are complementary rather than alternatives. - [ ] **Interrupt granularity in `sampleBurst`.** `interruptGenerate()` only sets a flag the caller's loop reads between `decode()` calls, and a burst runs all K forward steps without checking it — so a preempted job finishes its whole burst first (~583 ms at K=15). It binds in three cases: the seconds-long window while `#grow()` builds a second engine, a machine where `#growthBlocked` is set, and any third concurrent task once the pool is at cap. Fix in our `multistep.js`, not WebLLM; `discardLookahead` already handles the resulting state. - [ ] **Auto-tune `decodeSteps` from the burst probe.** K=15 is a constant derived from one machine and one model — Firefox on Apple Silicon, `Qwen3.5-0.8B` at ~7.3 ms/step. `multistep.js`'s own header says the best K falls as the model grows (a 25 ms/step model wants K=4), so the shipped default is wrong for every model but the one it was measured on, and wrong silently: a bad K costs throughput and reports nothing. **The infrastructure is already there.** `onBurst` delivers `{ steps, tokens, ms, encodeMs, syncMs, dispatches, forwardDispatches, flushes }` per burst ([multistep.js:246](src/engine/multistep.js#L246)), `setSteps()` is hot ([multistep.js:259](src/engine/multistep.js#L259)), and `WORKER_CONFIGURE` already calls `resetStats()` on every retune so "a sweep's points never bleed into each other" ([engine-worker.js:137](src/engine/engine-worker.js#L137)). It was built for this. **The obvious derivation does not work.** `K = floor(tickMs / perStepMs)` needs `perStepMs`, and the probe cannot separate it: `syncMs` is GPU execution *plus* the wait for the poll tick, which is the quantity being solved for. Only `encodeMs` is clean — the K-step loop contains no `await`, so it is content-process CPU alone — and it is the smaller half. **What works instead** is a hill-climb on the one quantity directly observed, `tokens / ms`. Throughput at K is `K / (ceil(K·p/tick)·tick)`; its peaks sit at `K = floor(n·tick/p)` and are asymptotically equal in n, so the first peak is the target and climbing K until throughput drops finds it without ever naming `p` or `tick`. That is also what makes it survive Chrome, where the 100 ms tick premise does not hold. **Three subtleties that will bite.** (1) `onBurst.steps` is the *clamped* burst size — `burstSize()` cuts it by `max_tokens` and the context window ([multistep.js:297](src/engine/multistep.js#L297)) — so the tuner must learn only from bursts where `steps === config.steps`, or it learns from end-of-generation stubs. (2) The first burst on a pipeline carries shader compilation; skip it. (3) `clampSteps("auto")` is **1** today (`Number("auto")` → NaN → `|| 1`), so an `"auto"` sentinel silently *disables* multi-step unless `clampSteps` and [API.md](API.md)'s documented `1`–`32` range change together — `api-doc.test.mjs` asserts the two agree. **Sequencing.** Composes with per-priority K below rather than replacing it: this finds the throughput ceiling K_max for the device+model, per-priority then spends it (`interactive` takes min(K_max, 4), `background` takes K_max). Do this one first — per-priority is written against the 15 this replaces. Unlike the rest of section 4 it touches **no new tvmjs internals**: it reads telemetry that already flows and calls a setter that already exists, which is the argument for lifting the freeze on this item alone. It cannot be *validated* here, though — the sawtooth only exists on real hardware, and Gate B (Chrome) is what proves the controller is finding a real peak rather than re-deriving 15. - [ ] **Per-priority `decodeSteps`.** K=15 maximises throughput but emits 15 tokens every ~583 ms, which reads as a stall. `interactive` should use K=2–4, `background`/`batch` K=32. Do this *after* the interrupt fix — raising K for background work lengthens exactly the bursts that preemption has to wait out. - [ ] **Restore cross-turn KV reuse.** Every turn re-prefills the whole history at **5.27 ms/token**, so a turn near the 4096 limit pays ~22 s before its first token. Pack `batch_prefill_paged_kv_kernel`'s six i32 metadata buffers into one with offsets (10 bindings → 5); the offsets already exist in its uniform block. ## 5. Test and infrastructure Both are "fix when it next bites", not scheduled work. The completed e2e run that used to sit here is in [ARCHIVE.md](ARCHIVE.md), including the `storageBuffersPerStage=9` anomaly worth re-checking on the next run. - [ ] Fix `PROFILE_PATH` in [test/e2e/run.mjs](test/e2e/run.mjs): it passes `--profile-path`, but web-ext 8 calls it `--firefox-profile` and exits with `Unknown arguments`. - [~] **Isolate the bench's pass-sweep onto its own device** — done, and it **did not do what the item assumed**. 2048 compute passes in one encoder loses the WebGPU device, and a lost device does not throw: later calls silently no-op. Two corrections from measuring it. First, the stated worry (later probes poisoned) was empty — the sweep was already the last phase, so nothing ran after it. The real damage was to the sweep's **own** numbers: the run that prompted this reported `perPass=-3.9us`, 512 passes measured faster than one, which is the device dying mid-sweep and the remaining submits becoming free. That half is fixed — `n512` now reads a plausible ~10us and `n2048` reports `discarded (device lost during the 2048-pass encode)` instead of inventing a number. Second, **isolation does not contain the loss.** With the sweep on its own device, `deviceLostDuringBench` still reports the main device lost during this phase, on both runs. A runaway command buffer on Metal appears to reset the whole adapter rather than one device on it. So the honest state is: sweep numbers are now trustworthy, containment is not achieved, and the bench is safe only because this is the last phase. Still open, if containment is wanted: run the sweep in a dedicated worker or a separate page so the reset cannot reach the measuring context at all. Also worth noting the attribution is imprecise — `lostDuring` records the phase at which `device.lost` *resolves*, and the 16384-dispatch probe immediately above is another plausible culprit. ## 6. Deferred, with the condition that would reopen them | Item | Reopens when | | --- | --- | | **Vision / image understanding** | An in-house compact VLM exists. The prebuilt option is `Phi-3.5-vision` at 3.95 GB, projecting ~4–5 tok/s and not co-resident with a text model in 16 GB. `modelType: "vlm"` already flows through registration, so the compile lands on an engine that accepts it. Open questions to answer against the real model: whether multi-step decoding survives an image prefill, and whether the Firefox 9-storage-buffer workaround holds for the vision tower's kernels. | | **MV3 migration** | `demo` is forced off MV2. `CreateExtensionServiceWorkerMLCEngine` solves the messaging and in-work keep-alive but **not** the actual problem — an idle service worker is killed and takes the resident multi-GB model with it. Use the native helper for messaging; hold the engine in an offscreen document, persistent page, or dedicated tab. Also verify WebGPU is exposed in a Chrome extension SW at all. | | **SRI / `verifyIntegrity`** | Self-hosted models are hardened, or corruption is actually observed. `ModelStore.verify()` checks key *presence*; `verifyIntegrity` checks *content* — different problems. Use the native one; do not hand-roll a hash check. | | **Multi-model via `reload([...])`** | Memory pressure shows up before scheduling contention does. Rejected for now: `reload()` is all-or-nothing, so adding a third model to `{A, B}` reloads A and B too (~51 s each). Additive residency is a hard requirement and only `#pools` provides it. |