PLAN.md

VRAM Calculator — v1 Plan

Pre-flight VRAM predictor + llama-server command-builder. Single-GPU and mixed-GPU. Hero use case: "I want 1M context on a model trained for 262k — give me the command and tell me it'll still be coherent."

This document is the spine agreed in the /grilling session. The grilling is closed; everything below is decided.

Objective

Maximize quality at a VRAM ceiling. Quality proxy = highest-bpw quant that fits. Floor = the 4-bit class (q4_0 / rocmfp4 / nvfp4, ~4.0+ bpw). Sub-4-bit quants (2-bit/3-bit family) are offered only on explicit user override and flagged "you asked for it." Backend-agnostic.

The fit loop (core)

target  = user_target_n_ctx
yarn    = auto_configure_yarn(training_ctx, target)   # rope_freq_scale
fit     = max_context(arch, floor_quant, gpus)       # hard VRAM ceiling
result  = min(target, fit)
if fit < target:  present "best you can do = fit, because <reason>"
if extension_ratio crosses ladder: warn (never hard-block)

Says "no" cleanly when impossible at floor + hardware, and offers the floor-quant max_context as the honest fallback.

YaRN coherence ladder (warnings only, never a hard block)

Monotonic — more extension always means same-or-more concern, never goes quiet:

• ≤ 2× extension — fine, no warning.
• > 2× up to 8× — "expect some quality degradation at the tail."
• ≥ 8× extension — "likely incoherent at long range; not recommended."

Source: YaRN paper behavior + community experience, not measured. Extension ratio = target_n_ctx / training_ctx (1.0 when target ≤ training or training unknown). Implemented as yarn_coherence_warnings(training_ctx, target_ctx) in yarn.py, surfaced in estimate() warnings and the UI.

Margin = the honesty knob

safety_margin_pct is the user-facing "how scared am I" knob. Complement path: the predictor emits fully explicit flags (-c, -ngl, -sm, -ts, -mg, -ctk, -ctv, --flash-attn, YaRN flags) plus --fit on --fit-target <margin MiB>. Because everything is already explicit, runtime --fit is a no-op safety net — the user gets exactly what the predictor said. Our number is the guarantee, with the margin as the dial. Never use the word "guarantee" in UI copy; the margin is the honest version.

Speculative decoding — two branches (this is a bugfix)

The shipped --spec-type draft is invalid. The real enum (from the llama.cpp server README) is:

none, draft-simple, draft-eagle3, draft-mtp, draft-dflash,
ngram-simple, ngram-map-k, ngram-map-k4v, ngram-mod, ngram-cache

Rewrite the spec surface to this enum, then split the add-on flow:

• Weightless (ngram-*): ~zero extra VRAM. Auto-offer when headroom exists; no "acknowledge the tradeoff" checkbox (there's no VRAM tradeoff).
• Weighted (draft-simple / draft-eagle3 / draft-mtp / draft-dflash): a draft model with weights + its own KV. Headroom-gated + an acknowledge checkbox. Draft rides the main GPU. Emit the full --spec-draft-* namespace (-md, --spec-draft-ngl, --spec-draft-device, --spec-draft-n-max, --spec-draft-p-min, --spec-draft-p-split), not the old -md-only stub.

draft-mtp (a spec type) is distinct from --mtp N (MTP head count on the target). Keep both but emit the right flag for each.

Backend scope

Dropped. Assume it runs. One displayed caveat: "fits in VRAM assuming your build supports the chosen quant." Keep the existing stock-build ROCmFP4/ROCmFPX incompatibility warning (that's not a backend issue — no stock build can read those tensor types at all).

Quarantined unverified data

Do not block v1 on these:

• nvfp4 (llama.cpp PR #22196): not yet in QUANT_BPW. Add a provisional row with a # VERIFY before ship comment once we have a verified bits-per-weight; until then it's absent from the table and can't be selected as a floor.
• dSpark (#25167 / #25173 / DeepSeek-V4-Flash-DSpark): folds into the existing draft-model VRAM path and emits --spec-type draft-mtp provisionally. For a predictor, dSpark's VRAM cost is just draft-weights + draft-KV, same as any MTP draft; the spec-type label only affects the command string.
• Vulkan quant matrix: dropped (consequence of dropping backends).

UI — two phases, two graphs

1. Fetch phase: repo ID + HF token → parse_hf_range once → cache ModelArch. Network happens here, once, never again.
2. Live phase: everything is estimate() recompute on the cached arch (microseconds). Context slider, quant selector, KV-dtype, GPU sizes, margin. The recommended command updates live.
3. Graph 1: VRAM vs n_ctx, one curve per quant — the context ceiling, with the slider sitting on it.
4. Graph 2: VRAM vs quant at fixed n_ctx, with quality as color/rank — the cost+quality frontier (where "maximize quality at the ceiling" becomes visible).

Implementation order

Done:

1. ✅ yarn.py — yarn_coherence_warnings(training_ctx, target_ctx) + extension_ratio; wired into yarn_warnings() (and thus estimate()).
2. ✅ report.py — emits --fit on --fit-target <margin MiB> via _fit_target_mib() (margin from smallest GPU's VRAM share).
3. ✅ draft.py + report.py — spec surface rewritten to the real llama.cpp enum (draft-simple, draft-eagle3, draft-mtp, draft-dflash, ngram-*); weightless (ngram, zero VRAM) vs weighted (draft, weights+KV) split; command_preview emits -md + the full --spec-draft-* namespace for weighted, only --spec-type for weightless.
4. ✅ quant.py — nvfp4 documented as a commented placeholder with a VERIFY note; not selectable as a floor until a verified bpw is sourced.
5. ✅ Tests — 67 passing (real-enum spec tests, YaRN coherence-ladder tests, --fit/--fit-target tests, weightless/weighted branch tests).
6. ✅ app.py — two-phase UI (fetch once → live estimate() recompute on every change), two graphs (VRAM-vs-n_ctx by quant; VRAM-vs-quant with quality color), the "assumes your build supports the chosen quant" caveat, and the sub-floor "you asked for it" warning. Syntax-checked; graph math verified against presets (gradio not installable locally).

Out of scope for v1

Per-backend quant compatibility matrices, MLX, Vulkan-specific quants, runtime auto-tuning (that's llama.cpp's --fit), measured perplexity/WikiText quality tables, throughput/token-per-second modeling.