# Performance benchmark pack A self-contained, portable harness for benchmarking the MVANet ONNX background-remover with **onnxruntime-node** across execution providers (**CPU / WebGPU / DirectML**) and input resolutions (**8K / 4K / 1080p / 512²**). It produces a plain-text report with a per-run timing breakdown plus RAM and **GPU memory (VRAM)** usage — and the VRAM probing works on **any GPU vendor, including Intel integrated graphics**, with no external tooling. It lives at [`../.tmp/performance/`](../.tmp/performance) (built to be copied wholesale to other machines). See its own [`README.md`](../.tmp/performance/README.md) for day-to-day usage; this doc is the *why/how* write-up, focused on the non-obvious parts. ## What it measures For every (execution provider × resolution) cell it runs **N sequential runs (default 15) with a fixed gap (default 5 s)** and records, per run: | column | meaning | |--------|---------| | `pre` | host preprocessing ms (sharp: decode + resize source → 1024², normalize) | | `infer` | `session.run()` ms (the ONNX graph) | | `post` | host postprocessing ms (logits → mask, resize mask back to W×H) | | `total` | pre + infer + post | | `ramFreeB` | system RAM free **before** the run | | `peakRSS` | peak Node process RSS **during** the run | | `vramFreeB` | GPU memory free **before** the run | | `peakVram` | peak GPU memory **used during** the run | Plus a per-cell summary (min / median / mean / max / p95) and the machine specs (CPU, GPU, RAM total, GPU memory total). DirectML init/run failures are caught and **recorded in the report** (error + stack) rather than aborting the run — on some driver/hardware combos DML fails, and the point is to capture that, not hide it. ### The static-input nuance (important when reading the report) The model input is a **static 1024×1024** tensor (Swin hard-codes it), so **ORT inference time is essentially constant across input resolutions**. The resolution dimension only moves `pre` (down- resizing an 8K source to 1024²) and `post` (up-resizing the 1024² mask back to the source size). The report breaks out `pre / infer / post` precisely so this is visible rather than confusing. ## GPU memory (VRAM) probing — vendor-agnostic, no `nvidia-smi` This is the non-obvious part worth documenting. The harness reads GPU memory through **one universal Windows path** — no `nvidia-smi`, no DXGI native code, no per-vendor branches, no flags. It is fully automatic and produces the same numbers Windows Task Manager shows. | quantity | source | |----------|--------| | **Used** (per adapter, system-wide) | `GPU Adapter Memory` performance counters: `Dedicated Usage` + `Shared Usage` (via `Get-Counter`) | | **Dedicated total** | registry `HKLM\…\Class\{4d36e968-…}\NNNN\HardwareInformation.qwMemorySize` | | **Shared limit** | ≈ 50 % of system RAM (the WDDM default — Task Manager's "Shared GPU memory") | with ``` total = dedicated total + shared limit used = dedicated used + shared used (of the busiest adapter) free = total - used ``` ### Why no `nvidia-smi` It was verified on an NVIDIA box that the registry `qwMemorySize` reads **16303 MB — exactly matching `nvidia-smi`'s reported total**, and the `Dedicated Usage` perf counter matches `nvidia-smi`'s *used* to within a few hundred MB (driver-reserve difference). So the registry + perf-counter pair reproduces `nvidia-smi` for free, on every vendor, with no external dependency. Dropping it removed the only per-vendor branch and the only fallback path. ### Why this works for Intel integrated graphics Intel iGPUs have **~no dedicated VRAM** — a tiny dedicated carve-out plus a large **shared** pool taken from system RAM. The same formula describes them correctly: their `dedicated total` is small, so the **shared pool is the real budget**, and `used`/`free` come from the same counters. This is exactly how Task Manager presents an iGPU's "GPU memory". `GPU Adapter Memory` counters exist on Windows 10 1709+ for any WDDM 2.x driver, so the path is essentially universal on modern Windows. If GPU performance counters aren't present, the report says so and skips VRAM sampling instead of failing. ## Gotchas (hard-won, specific to this harness) - **`Get-Counter` adapter instances are LUID-keyed** (`luid_0x…_phys_0`), not names. The registry class keys give names + `qwMemorySize` but **no LUID**, so the two can't be perfectly joined without DXGI. The harness sidesteps it: the **busiest adapter** (highest dedicated+shared usage) is the one doing the work, and `dedicated total` = the max `qwMemorySize` across adapters. Correct for single-GPU machines and for hybrid laptops where the EP uses the discrete GPU (the usual case). - **PowerShell exits code 1 while still printing valid JSON.** The registry query trips a suppressed non-terminating error that sets `$LASTEXITCODE=1` even though stdout is correct. `execFile` rejects on non-zero exit, so `tryExec` was discarding good output → the fix **salvages stdout on a non-zero exit** (only a truly absent tool / empty stdout → `null`). - **Short GPU runs would report `peakVram = n/a`.** A single `Get-Counter` spawn takes ~0.5–0.9 s under GPU load, so a ~1.3 s WebGPU/512 run could finish before any sample completed. Fix: the peak sampler is **seeded with the before-run sample**. Since VRAM here is dominated by resident model weights (allocated at session creation, before the run), that seed is an accurate estimate, not just a floor. - **onnxruntime-node runs inference on a background thread**, so the JS event loop stays free and the `setInterval` peak sampler actually fires during `session.run()`. (Perf-counter sampling spawns PowerShell, so it polls at ~1.4 Hz — `intervalMs: 700` — with a `_busy` guard against overlap.) - **ORT constant-fold warnings** ("Could not find a CPU kernel…") are harmless stderr noise at session creation; quieted with `ort.env.logLevel='error'` + `logSeverityLevel: 3`. They never entered the report (stderr only). ## Honest caveats - The **shared limit is an estimate** (50 % of RAM, the Windows default). Intel's newer "Shared GPU Memory Override" can raise it, so on such a machine the *total* may read low; **`used` and `dedicated total` stay exact**. The report labels it "(≈50 % RAM)". - VRAM accounting is **Task-Manager-style (dedicated + shared)** by design. For a discrete card the headline `total` is therefore dedicated + shared (e.g. 16 GB + 16 GB = 32 GB), not dedicated-only. ## Files ``` .tmp/performance/ benchmark.mjs main harness -> report-.txt make-bench-images.mjs generates the 4 test images into images/ sysinfo.mjs CPU/GPU/RAM/VRAM probing + the peak sampler (createGpuProbe, PeakSampler) package.json run.ps1 deps + one-shot wrapper (install -> images -> bench) README.md usage on a fresh machine models/ drop the fp32 .onnx here (ships with mvanet_box_segmenter.onnx) images/ node_modules/ generated images; pre-copied Windows-x64 deps SAMPLE-report-1run-per-cell.txt example output (1 run/cell across all EPs) ``` ## Reference - `IDXGIAdapter3::QueryVideoMemoryInfo` (the native equivalent we deliberately avoid): - Intel "Shared GPU Memory Override" (why the shared limit is an estimate):