docs(roadmap): expose worker packaging defect as an independent item in section 1

3c72141 verified 14 days ago

16 kB

Roadmap — open work only

This is the only list of open work. Anything finished moves to ARCHIVE.md with the reasoning that produced it. AI.md is the reference for what is true and measured; API.md is the full call surface, asserted against the code; WEBLLM-SURFACE.md is what WebLLM already does and must be read before adding a capability.

Sections are named, not numbered. An earlier split across three files used "Track 1"/"Track 2" to mean different things in each, which is how a reader ends up implementing the wrong item.

Sequencing. The whole project exists to make WebLLM more compatible, easier to use, foolproof to build on. Every item below is ranked against that. everything-webgpu@0.1.0 is on npm as of this writing — the install path, the four-line demo, the ergonomic verbs, API.md and examples/ all shipped and are recorded in ARCHIVE.md. Section 1 is what remains before the extraction is fully proven: the extension itself rebuilding on the package. Section 4 (performance) is frozen as a block, except for "Measure Ollama", until a real user calls it slow: it is the highest-risk, highest-effort work and it re-touches the tvmjs internals the WebLLM-upgrade section (also in ARCHIVE.md) just spent its time hardening.

1. Ship it — the last mile

everything-webgpu@0.1.0 is on npm. What shipped — the install path, the four-line demo, the ergonomic verbs, API.md, examples/, the bundle-size story, licence compliance — is in ARCHIVE.md, "Shipping 0.1.0". What remains is proving the boundary from the other side.

demo consumes the package, and the popup + manager leave main. The acceptance test for the whole extraction: if the extension rebuilds on the published package, the boundary is right. npm run e2e passing is the same claim at the source-tree level.
[~] Gate A — a bare Vite page loads a prebuilt model and generates. examples/bare/ exists and the GPU-free half is proven: vite build resolves through the exports map and splits into an entry chunk, the decode worker as its own chunk, and the 6 MB WebLLM bundle as a lazy one. What remains is one run in a WebGPU browser to confirm it generates. npm run verify-consumer already exercises a real install end to end, short of the actual generation.
Defect: Worker unbundled relative imports in production (vite build / Rollup blind spot). First real-browser run on a static host (Hugging Face Spaces) surfaced a packaging defect that passed all prior headless assertions. On vite build, engine.load() threw PACKAGE_INCOMPLETE: "The decode worker failed to load from .../assets/engine-worker-.js"*. Root cause: Rollup's new Worker(new URL(..., import.meta.url), { type: "module" }) handling copies engine-worker.js as an asset chunk without bundling or rewriting its internal ESM imports. The emitted worker still carries: import { WebWorkerMLCEngineHandler } from "../../vendor/web-llm.js" import { WORKER_CONFIGURE } from "./constants.js" import { DEFAULT_DECODE_STEPS } from "./multistep.js" On a static production deployment, these relative URLs resolve to 404s on the root domain, killing the worker silently before handshake. vite dev hid this because optimizeDeps.exclude served files directly from source where siblings exist. test/consumer/verify.mjs line 130 missed it because it asserted file existence (assets.some(f => f.startsWith("engine-worker"))), never executing the worker in a browser. Resolution: - [x] Demo workaround: Pre-bundle engine-worker.js with esbuild --bundle --format=esm into a standalone 6.58 MB file (public/engine-worker.js) with zero external imports, and pass explicit workerUrl: new URL("engine-worker.js", window.location.href). - [ ] Library permanent fix for 0.1.1: In build.mjs, pre-bundle engine-worker.js into a self-contained dist/worker.bundle.js at package build time. Point DEFAULT_WORKER_URL at the bundle so zero consumer configuration or workerUrl is required. Update verify-consumer with a headless browser fetch/execute assertion to prevent regressions.
Gate A′ — the same page ingests a local folder. cache.put() against the synthetic local-model.invalid key is proven only on an extension origin. Gates the offline route, not the library.
Gate B — Chrome. Measure tok/s. Expected to beat Firefox because KV reuse is not disabled there. Entirely a prediction today; probeDevice() is the instrument that makes it reportable.
Every ERROR code carries an actionable fix. The preflight (environment()) and the typed errors (9 codes now, PACKAGE_INCOMPLETE included) both exist; what is not yet done is the systematic pass confirming each code's message/detail names a cause and a fix, and that the README/API.md point at environment() as the "will this run here?" call it is meant to be.

2. Verb consolidation — done

load(src, opts), unload(id, level) / remove() / unloadAll(), and environment() + environment.measure() all shipped. The reasoning — the two dropped load() dispatch rules, why a bare unload() frees only the current model, why read and write are split — is in ARCHIVE.md, "Verb consolidation". chat.completions.create() is unchanged and stays that way.

3. Engine capability

prefetch(), embed() / embedRaw(), and the ask() / conversation() / ghostText() recipes shipped — see ARCHIVE.md, "Engine capability". One item is still open:

LRU eviction when quota is short. cacheState() and evict() exist; nothing yet decides what to drop. Table stakes for "it just works" the moment a second model is cached — but the eviction policy is a decision, not an implementation detail.

4. Performance — frozen, except the measurement

Model- and kernel-level work, independent of the library structure — and therefore independent of everything this project is for. Frozen as a block until a real user calls it slow. The library ships now, so "nobody can install it" is no longer the argument; the argument is that this is the highest-risk, highest-effort work on the page, it re-touches the tvmjs internals the WebLLM-upgrade section (in ARCHIVE.md) just spent its time hardening, and a speedup nobody has asked for is risk without a return. The one exception is "Measure Ollama": it is cheap, and it decides whether any of the rest is worth its risk.

Retune the dlight GEMV schedule — more work per thread before the reduction. Measured 1.83x in isolation, which would put decode near the ~41 GB/s dequant ceiling. Confirmed untaken on the 2B build. Needs a recompile; toolchain is stood up.
Batched decode. The model lib exports batch_decode / batch_prefill / batch_verify and a paged KV cache; WebLLM hardcodes defaultMaxNumSequence = 1 and numSamples = 1 (bundle lines 15250, 15276). Lifting that reads the weights once per step for N sequences — projected ~4x on the batch API, which is the page-translation shape and the one workload the engine currently makes no faster than a loop of chat calls. No recompilation needed. Not a fix for the 1.06x second-engine measurement: a second engine buys task isolation, never aggregate throughput, and the two are complementary rather than alternatives.
Interrupt granularity in sampleBurst. interruptGenerate() only sets a flag the caller's loop reads between decode() calls, and a burst runs all K forward steps without checking it — so a preempted job finishes its whole burst first (~583 ms at K=15). It binds in three cases: the seconds-long window while #grow() builds a second engine, a machine where #growthBlocked is set, and any third concurrent task once the pool is at cap. Fix in our multistep.js, not WebLLM; discardLookahead already handles the resulting state.
Auto-tune decodeSteps from the burst probe. K=15 is a constant derived from one machine and one model — Firefox on Apple Silicon, Qwen3.5-0.8B at ~7.3 ms/step. multistep.js's own header says the best K falls as the model grows (a 25 ms/step model wants K=4), so the shipped default is wrong for every model but the one it was measured on, and wrong silently: a bad K costs throughput and reports nothing. The infrastructure is already there. onBurst delivers { steps, tokens, ms, encodeMs, syncMs, dispatches, forwardDispatches, flushes } per burst (multistep.js:246), setSteps() is hot (multistep.js:259), and WORKER_CONFIGURE already calls resetStats() on every retune so "a sweep's points never bleed into each other" (engine-worker.js:137). It was built for this. The obvious derivation does not work. K = floor(tickMs / perStepMs) needs perStepMs, and the probe cannot separate it: syncMs is GPU execution plus the wait for the poll tick, which is the quantity being solved for. Only encodeMs is clean — the K-step loop contains no await, so it is content-process CPU alone — and it is the smaller half. What works instead is a hill-climb on the one quantity directly observed, tokens / ms. Throughput at K is K / (ceil(K·p/tick)·tick); its peaks sit at K = floor(n·tick/p) and are asymptotically equal in n, so the first peak is the target and climbing K until throughput drops finds it without ever naming p or tick. That is also what makes it survive Chrome, where the 100 ms tick premise does not hold. Three subtleties that will bite. (1) onBurst.steps is the clamped burst size — burstSize() cuts it by max_tokens and the context window (multistep.js:297) — so the tuner must learn only from bursts where steps === config.steps, or it learns from end-of-generation stubs. (2) The first burst on a pipeline carries shader compilation; skip it. (3) clampSteps("auto") is 1 today (Number("auto") → NaN → || 1), so an "auto" sentinel silently disables multi-step unless clampSteps and API.md's documented 1–32 range change together — api-doc.test.mjs asserts the two agree. Sequencing. Composes with per-priority K below rather than replacing it: this finds the throughput ceiling K_max for the device+model, per-priority then spends it (interactive takes min(K_max, 4), background takes K_max). Do this one first — per-priority is written against the 15 this replaces. Unlike the rest of section 4 it touches no new tvmjs internals: it reads telemetry that already flows and calls a setter that already exists, which is the argument for lifting the freeze on this item alone. It cannot be validated here, though — the sawtooth only exists on real hardware, and Gate B (Chrome) is what proves the controller is finding a real peak rather than re-deriving 15.
Per-priority decodeSteps. K=15 maximises throughput but emits 15 tokens every ~583 ms, which reads as a stall. interactive should use K=2–4, background/batch K=32. Do this after the interrupt fix — raising K for background work lengthens exactly the bursts that preemption has to wait out.
Restore cross-turn KV reuse. Every turn re-prefills the whole history at 5.27 ms/token, so a turn near the 4096 limit pays ~22 s before its first token. Pack batch_prefill_paged_kv_kernel's six i32 metadata buffers into one with offsets (10 bindings → 5); the offsets already exist in its uniform block.

5. Test and infrastructure

Both are "fix when it next bites", not scheduled work. The completed e2e run that used to sit here is in ARCHIVE.md, including the storageBuffersPerStage=9 anomaly worth re-checking on the next run.

Fix PROFILE_PATH in test/e2e/run.mjs: it passes --profile-path, but web-ext 8 calls it --firefox-profile and exits with Unknown arguments.
[~] Isolate the bench's pass-sweep onto its own device — done, and it did not do what the item assumed. 2048 compute passes in one encoder loses the WebGPU device, and a lost device does not throw: later calls silently no-op. Two corrections from measuring it. First, the stated worry (later probes poisoned) was empty — the sweep was already the last phase, so nothing ran after it. The real damage was to the sweep's own numbers: the run that prompted this reported perPass=-3.9us, 512 passes measured faster than one, which is the device dying mid-sweep and the remaining submits becoming free. That half is fixed — n512 now reads a plausible ~10us and n2048 reports discarded (device lost during the 2048-pass encode) instead of inventing a number. Second, isolation does not contain the loss. With the sweep on its own device, deviceLostDuringBench still reports the main device lost during this phase, on both runs. A runaway command buffer on Metal appears to reset the whole adapter rather than one device on it. So the honest state is: sweep numbers are now trustworthy, containment is not achieved, and the bench is safe only because this is the last phase. Still open, if containment is wanted: run the sweep in a dedicated worker or a separate page so the reset cannot reach the measuring context at all. Also worth noting the attribution is imprecise — lostDuring records the phase at which device.lost resolves, and the 16384-dispatch probe immediately above is another plausible culprit.

6. Deferred, with the condition that would reopen them

Item	Reopens when
Vision / image understanding	An in-house compact VLM exists. The prebuilt option is `Phi-3.5-vision` at 3.95 GB, projecting ~4–5 tok/s and not co-resident with a text model in 16 GB. `modelType: "vlm"` already flows through registration, so the compile lands on an engine that accepts it. Open questions to answer against the real model: whether multi-step decoding survives an image prefill, and whether the Firefox 9-storage-buffer workaround holds for the vision tower's kernels.
MV3 migration	`demo` is forced off MV2. `CreateExtensionServiceWorkerMLCEngine` solves the messaging and in-work keep-alive but not the actual problem — an idle service worker is killed and takes the resident multi-GB model with it. Use the native helper for messaging; hold the engine in an offscreen document, persistent page, or dedicated tab. Also verify WebGPU is exposed in a Chrome extension SW at all.
SRI / `verifyIntegrity`	Self-hosted models are hardened, or corruption is actually observed. `ModelStore.verify()` checks key presence; `verifyIntegrity` checks content — different problems. Use the native one; do not hand-roll a hash check.
Multi-model via `reload([...])`	Memory pressure shows up before scheduling contention does. Rejected for now: `reload()` is all-or-nothing, so adding a third model to `{A, B}` reloads A and B too (~51 s each). Additive residency is a hard requirement and only `#pools` provides it.