---
license: mit
library_name: transformers.js
pipeline_tag: mask-generation
base_model:
- finegrain/finegrain-box-segmenter
base_model_relation: quantized
tags:
- onnx
- onnxruntime
- onnxruntime-web
- webgpu
- vision
- image-segmentation
- mask-generation
- background-removal
- salient-object-detection
- matting
- mvanet
---
# finegrain-box-segmenter — ONNX
FP32 ONNX export of [**finegrain/finegrain-box-segmenter**](https://huggingface.co/finegrain/finegrain-box-segmenter) (an MVANet high-resolution background remover), ready to run in the browser, Node.js and Python with [ONNX Runtime](https://onnxruntime.ai/) (CPU, WebGPU (recommended), DirectML).
- **Base model:** `finegrain/finegrain-box-segmenter` (MIT) — MVANet, SafeTensors, [arXiv:2404.07445](https://arxiv.org/abs/2404.07445)
- **Architecture:** MVANet with a **Swin-B** backbone, **\~94.6 M parameters**; static 1024×1024
input, batch fixed at 1.
- **What this repo adds:** the same network exported to ONNX (full-precision floating-point **fp32**), so it runs
without PyTorch/refiners.
- **Why fp32:** maximum compatibility with zero precision risk — the fp32 graph runs on CPU, WebGPU and DirectML, and stays **bit-identical** to the original release; at \~94.6 M parameters it is still cheap enough to run on a local machine. (An fp16 build (\~403 MB) exists and its masks matched fp32 in testing, but it is not published, to keep the release simple — open a discussion if you need it.) Regarding why this release uses the "quantized" tag: it's a fairly common convention for ONNX / format derivatives (same as e.g. `onnx-community/BiRefNet-ONNX`), so I decided to go this route for tagging.
- **Output is bit-identical to the PyTorch reference** on CPU (mask MAE `0.000 / 255`; random-input logits `max|Δ| = 1.5e-5`).
## Examples
These tests are **intentionally made to be difficult** for the models, doing their best to expose the weak points of each model.
1. Sub-pixel hair strands
2. Refraction + small object touching frame edge
3. Large shape touching edge in a style rarely seen in the training data (impasto), no obvious dominant subject
4. 3D/miniature with glows and object blending under water
5. Cartoon / flat 2D
6. Complex fire pattern
7. Reflections on a confusing background
8. Intricate many-holed topology
The short version: **finegrain-box-segmenter is strongest on soft, semi-transparent edges** — it keeps more sub-pixel hair strands (1), cleaner glass/refraction edges (2) and more of the flame and fur tips (6) than the other tools, and it resolves the many-holed branch topology (8) cleanly. Its weak spots in this set: the ultra-thin rigging wires in (4) (the BiRefNet variants preserve them better) and the unusual impasto style of (3), where part of the rooster's comb goes semi-transparent. All cutouts were produced the same way: each tool's standard whole-image pipeline — no boxes or prompts — on identical source images, and every crop is the same pixel window. The competitors (`finegrain` is this model): `ben2` = [BEN2](https://huggingface.co/PramaLLC/BEN2), `bria` = [BRIA RMBG-2.0](https://huggingface.co/briaai/RMBG-2.0), `birefnet` = [BiRefNet](https://huggingface.co/ZhengPeng7/BiRefNet-DIS5K-TR_TEs), the massive-trained checkpoint (`BiRefNet-massive-TR_DIS5K_TR_TEs-epoch_420`, run via [rembg](https://github.com/danielgatis/rembg)).
| Original | finegrain | ben2 | bria | birefnet |
|---|
| 1 |  |  |  |  |  |
|---|
 |  |  |  |  |
| 2 |  |  |  |  |  |
|---|
 |  |  |  |  |
| 3 |  |  |  |  |  |
|---|
 |  |  |  |  |
| 4 |  |  |  |  |  |
|---|
 |  |  |  |  |
| 5 |  |  |  |  |  |
|---|
 |  |  |  |  |
| 6 |  |  |  |  |  |
|---|
 |  |  |  |  |
| 7 |  |  |  |  |  |
|---|
 |  |  |  |  |
| 8 |  |  |  |  |  |
|---|
 |  |  |  |  |
## Files
| File | Precision | Size | Notes |
|------|-----------|-----:|-------|
| `onnx/model.onnx` | fp32 | \~805 MB | full precision; CPU + WebGPU; bit-identical parity with PyTorch |
ONNX operator set version 17, \~94.6 M parameters, SHA256(onnx/model.onnx) = `d3f60fc778f2b14435a8df3d61511c293cec3a88c670ad8ecac5035c6b9730d5`
> Why 805 MB for a 94.6 M-param model: \~378 MB is fp32 weights; the other \~425 MB is constant-folded shifted-window (Swin) attention masks / position embeddings baked in at export (`do_constant_folding=True`). Note that disabling the folding will make the file *larger*, not smaller.
## I/O contract
| | name | shape | dtype |
|---|---|---|---|
| input | `input` | `[1, 3, 1024, 1024]` | float32 |
| output | `logits` | `[1, 1, 1024, 1024]` | float32 (raw logits) |
**Pre-processing** (must be reproduced by the caller): take the RGB image, resize to 1024×1024 (bilinear or a comparable resampler — mild resampler differences don't visibly change the mask), scale to `[0,1]` (`/255`), normalize with ImageNet statistics `mean = [0.485, 0.456, 0.406]`, `std = [0.229, 0.224, 0.225]`, layout NCHW.
**Post-processing:** apply **`sigmoid`** to the logits to get an alpha matte in `[0,1]`, then resize it back to the original image size. Use it directly as a mask, or as the alpha channel of an RGBA cutout. (Note: this model emits **raw logits** — apply sigmoid; do *not* min-max normalize the output the way some other ONNX matting models, e.g. BEN2, require.)
## How it was converted
Exported from the published [v0.1 SafeTensors weights](https://huggingface.co/finegrain/finegrain-box-segmenter) with PyTorch's **legacy TorchScript exporter** (`torch.onnx.export` with the dynamo path disabled) at opset 17, with constant folding enabled. The raw `nn.Module` — obtained from refiners' `BoxSegmenter` and put in eval / float mode — is traced on a single `1×3×1024×1024` float input, and its tensors are named `input` and `logits`. There are no architecture changes and no retraining: the exported graph is exactly `image → logits`.
Two non-obvious choices were required:
- **Legacy exporter, not dynamo.** torch's dynamo / `torch.export` path fails decomposing Swin's `transpose(1,2).reshape(...)` (it lowers the non-contiguous view to a strict `aten.view` that can't represent the transposed strides). The legacy TorchScript tracer records a real `Reshape` and exports cleanly.
- `adaptive_avg_pool2d` → `avg_pool2d`. Refiners' pooling always divides evenly (`adaptive_avg_pool2d(x, (h//r, w//r))`), so it is **numerically identical** to a plain `avg_pool2d(kernel=r, stride=r)` that maps to ONNX `AveragePool` — an exact swap, not an approximation.
The graph uses only standard ops (`Conv, AveragePool, MatMul, Gemm, Resize, Softmax, LayerNormalization, PRelu, Add, Mul, Concat, Split, Slice, Reshape, Transpose`) — no `grid_sample` / `einsum` / `unfold` — so it runs on ORT CPU and WebGPU as-is.
**Parity** (`verify_parity.py`, CPU fp32): random input → torch vs ORT logits `max|Δ| = 1.5e-5` (sigmoid `3e-6`); refiners' golden cactus image → mask **MAE = 0.000 / 255** (bit-identical). The full sourced write-up lives in this repo at [`docs/HOW-CONVERSION-WAS-MADE.md`](docs/HOW-CONVERSION-WAS-MADE.md); to reproduce the export yourself, see [`docs/DEVELOPMENT.md`](docs/DEVELOPMENT.md) and the [`python/`](python) scripts.
## Quality — measured on *this* ONNX
The base model's published benchmark was re-run **on this ONNX export**: the full [`finegrain/finegrain-product-masks-lite`](https://huggingface.co/datasets/finegrain/finegrain-product-masks-lite) set (120 masks), scored with [PySODMetrics](https://github.com/lartpang/PySODMetrics). It was scored in two modes, which measure different things — don't mix them up when comparing models:
| Mode (same 120 masks) | MAE ↓ | S-measure ↑ | E-measure (mean) ↑ | Dice (mean) ↑ |
|---|--:|--:|--:|--:|
| **Box-prompted crop** — the base model's published protocol | **0.0079** | **0.9738** | **0.9854** | **0.9669** |
| **Whole-image (no box)** — how this release is used | 0.086 | 0.796 | 0.793 | 0.712 |
**Box-prompted crop mode** confirms the conversion: bit-identical parity implies identical task quality, and running the base model's own protocol on this ONNX reproduces its published numbers to within rounding:
| Metric | **This ONNX** | Published (base) | Δ |
|---|--:|--:|--:|
| MAE ↓ | **0.0079** | 0.0078 | +0.0001 |
| S-measure ↑ | **0.9738** | 0.974 | −0.0002 |
| E-measure (mean) ↑ | **0.9854** | 0.985 | +0.0004 |
| Dice (mean) ↑ | **0.9669** | 0.967 | −0.0001 |
n = 120. The sub-0.001 deltas are host-side resize / box-derivation rounding, **not** conversion error — the ONNX **inherits the original's quality** (which, on this set, beats `briaai/RMBG-1.4` and box-guided `ZhengPeng7/BiRefNet`; see the base card).
**Whole-image mode (no box)** scores far lower *by construction*: this dataset's ground truth is one *specific boxed object*, so scoring the no-prompt salient output against it penalizes the model whenever the image's salient region isn't exactly that boxed target. It's a worst-case lower bound on a box-oriented set — not representative of ordinary single-subject background removal (see the [Examples](#examples) above for what whole-image output actually looks like).
## Usage — Transformers.js (easiest)
This repo ships the standard `onnx/model.onnx` + `config.json` + `preprocessor_config.json` layout, so [Transformers.js](https://huggingface.co/docs/transformers.js) drives the whole pipeline for you — pre-process, sigmoid, and resize-back included:
```js
import { pipeline } from '@huggingface/transformers';
const segmenter = await pipeline('background-removal', 'MarcinEU/finegrain-box-segmenter-ONNX');
const out = await segmenter(['https://example.com/photo.jpg']);
out[0].save('cutout.png'); // RawImage RGBA - alpha is the matte
// out[0].toCanvas() / await out[0].toBlob() if you'd rather not save to disk
```
Verified end-to-end (`@huggingface/transformers` v4.2.0): the model resolves to `SwinForSemanticSegmentation`, Transformers.js remaps the processor output onto our `input` tensor and auto-applies `sigmoid` to the raw `logits` — no custom code. Add `{ device: 'webgpu' }` for the GPU. Use **v4.2.0 or newer** (the automatic input-name remap and auto-sigmoid are required). If you need full control over pre/post-processing, use the hand-rolled paths below.
## Usage — onnxruntime-web (browser, WebGPU)
```js
import * as ort from 'onnxruntime-web/webgpu';
const SIZE = 1024, MEAN = [0.485,0.456,0.406], STD = [0.229,0.224,0.225];
const session = await ort.InferenceSession.create(
'https://huggingface.co/MarcinEU/finegrain-box-segmenter-ONNX/resolve/main/onnx/model.onnx',
{ executionProviders: ['webgpu'] });
// draw your image into a 1024x1024 canvas, then:
const { data } = ctx.getImageData(0, 0, SIZE, SIZE); // RGBA, Uint8ClampedArray
const plane = SIZE * SIZE, chw = new Float32Array(3 * plane);
for (let p = 0; p < plane; p++) {
chw[p] = (data[p*4] / 255 - MEAN[0]) / STD[0];
chw[plane + p] = (data[p*4+1] / 255 - MEAN[1]) / STD[1];
chw[2*plane + p] = (data[p*4+2] / 255 - MEAN[2]) / STD[2];
}
const out = await session.run({ input: new ort.Tensor('float32', chw, [1,3,SIZE,SIZE]) });
const logits = out.logits.data; // Float32Array, 1024*1024
const alpha = logits.map(v => 1 / (1 + Math.exp(-v))); // sigmoid -> [0,1]
// resize `alpha` back to your image size and composite as the alpha channel.
```
## Usage — onnxruntime-node (Node.js, with `sharp`)
A complete, runnable background remover is in [`usage/remove_bg.mjs`](usage/remove_bg.mjs) — run it from the repo root (with `onnx/model.onnx` downloaded):
```bash
npm i onnxruntime-node sharp
node usage/remove_bg.mjs --image photo.jpg --model onnx/model.onnx # CPU
node usage/remove_bg.mjs --image photo.jpg --model onnx/model.onnx --ep webgpu # GPU (often 3-10x faster; see Performance below)
```
It resizes the whole image to 1024², normalizes, runs the session, applies sigmoid, resizes the mask back, and writes `*_mask.png` and `*_cutout.png` (RGBA). Outputs carry real transparency — view them over a checkerboard/solid background.
## Usage — Python (onnxruntime)
Verified against this exact export (`pip install onnxruntime pillow numpy huggingface_hub`); masks match the Node reference pipeline to \~1/255 mean difference:
```python
from huggingface_hub import hf_hub_download
from PIL import Image
import numpy as np, onnxruntime as ort
model_path = hf_hub_download("MarcinEU/finegrain-box-segmenter-ONNX", "onnx/model.onnx")
session = ort.InferenceSession(model_path)
img = Image.open("photo.jpg").convert("RGB")
x = np.asarray(img.resize((1024, 1024), Image.BILINEAR), dtype=np.float32) / 255.0
x = ((x - [0.485, 0.456, 0.406]) / [0.229, 0.224, 0.225]).transpose(2, 0, 1)[None].astype(np.float32)
logits = session.run(None, {"input": x})[0][0, 0]
alpha = 1.0 / (1.0 + np.exp(-logits)) # sigmoid -> [0,1]
mask = Image.fromarray((alpha * 255).astype(np.uint8)).resize(img.size, Image.BILINEAR)
cutout = img.copy(); cutout.putalpha(mask)
cutout.save("cutout.png"); mask.save("mask.png")
```
## Performance across environments
The graph is a **static 1024×1024 fp32** Swin-B (MVANet) running at internal **batch-5**, so it is **compute-bound and the inference time is \~constant regardless of input image size** — the source resolution only changes the negligible host-side resize (`pre` + `post`, ≈30–500 ms for 1080p→8K). The numbers below are therefore "**per image**" for any resolution. All EPs run the **same fp32 model** (no fp16/NPU path). Measured with `onnxruntime-node` 1.26, **15 runs/cell, median**.
### Measured — median inference time, seconds per image (lower is better)
| Machine (form factor) | CPU (cores/threads) | GPU | RAM | **CPU EP** | **WebGPU** | **DirectML** |
|---|---|---|--:|--:|--:|--:|
| Desktop + discrete GPU | i7-8700K (6C/12T) | **NVIDIA RTX 5070 Ti** 16 GB | 32 GB | 9.7 | **0.9** ⭐ | 1.5 |
| Laptop (Arrow Lake-H) | Core Ultra 7 255H (16C/16T) | Intel **Arc 140T** iGPU | 48 GB | **4.6** | 8.6 | **1.6** ⭐ |
| Laptop (Tiger Lake) | i7-1165G7 (4C/8T) | Intel Iris Xe 96 EU iGPU | 16 GB | 13.9 | **9.6** ⭐ | ✗ OOM at init |
| Laptop (Raptor Lake-U) | i5-1335U (10C/12T, 15 W) | Intel Iris Xe 80 EU iGPU | 8 GB | **≈15** ⭐ | ✗ OOM \~3934 MB¹ | ✗ device-hung |
⭐ = fastest working backend on that machine. ✗ = execution provider failed (footnotes ¹–³ below give the byte-level cause). Numbers here are the steady-state median across sizes; the **per-size mean ± std** is in the Reference anchors tables below. Add **pre+post** for end-to-end total: ≈0.05 s at 512²/1080p, ≈0.5 s at 8K.
**Neither the NPU nor any other accelerator is involved.** `onnxruntime-node` has no NPU EP here: the **CPU EP** uses CPU cores, **DirectML/WebGPU** use the **GPU**. The Core Ultra 7 255H's "Intel AI Boost" NPU is *not* used — its NPU TOPS are irrelevant to these numbers; the iGPU figures above are the **Arc 140T graphics**, not the NPU.
### Reference anchors — match your own machine
Every **hardware × EP combination measured**, broken out by input size. Cells are **mean ± std** of the per-image **inference** time in seconds (the warm-up run is dropped, so n ≈ 14; n = 10 where noted). ✗ = the EP failed — see footnotes. Because inference is \~size-independent (static 1024² input) a row's four cells are nearly equal; the size axis mainly exposes **where memory runs out** (e.g. the 8 GB laptop clears 8K but not the smaller sizes). fp16 TFLOPS are a published **proxy** for ranking GPUs (the model runs fp32, so real rates are lower); discrete-GPU bandwidth is on-card VRAM, iGPU bandwidth is the shared-RAM `memcpy` rate measured. `″` = same as the row above; the Arc 140T additionally has XMX matrix engines (74 INT8 TOPS) that its fp32 path does not use.
**GPU backends** (anchor = fp16 throughput + memory bandwidth):
| GPU (arch) | EP | fp16 (proxy) | Mem BW | 8K (s) | 4K (s) | 1080p (s) | 512² (s) |
|---|---|--:|--:|--:|--:|--:|--:|
| RTX 5070 Ti (Blackwell), discrete 16 GB | WebGPU | 88 TFLOPS | 896 GB/s | 0.95 ± 0.07 | 0.85 ± 0.06 | 0.85 ± 0.07 | 0.89 ± 0.08 |
| RTX 5070 Ti | DirectML | ″ | ″ | 1.50 ± 0.03 | 1.50 ± 0.06 | 1.48 ± 0.04 | 1.50 ± 0.04 |
| Arc 140T iGPU (Xe-LPG, 8 Xe @2.25 GHz) | DirectML | \~9 TFLOPS | \~39 GB/s sh. | 1.58 ± 0.02 | 1.57 ± 0.00 | 1.57 ± 0.00 | 1.57 ± 0.00 |
| Arc 140T | WebGPU | ″ | ″ | 8.61 ± 0.05 | 8.64 ± 0.08 | 8.64 ± 0.09 | 8.65 ± 0.07 |
| Iris Xe 96 EU iGPU (Tiger Lake) | WebGPU | 3.4 TFLOPS | \~31 GB/s sh. | 9.79 ± 0.40 | 9.52 ± 0.34 | 9.46 ± 0.25 | 9.77 ± 0.46 |
| Iris Xe 96 EU | DirectML | ″ | ″ | ✗² | ✗² | ✗² | ✗² |
| Iris Xe 80 EU iGPU (Raptor Lake-U) | WebGPU | \~3 TFLOPS | \~29 GB/s sh. | 33.4 ± 2.8¹ | ✗¹ | ✗¹ | ✗¹ |
| Iris Xe 80 EU | DirectML | ″ | ″ | ✗³ | ✗³ | ✗³ | ✗³ |
¹ **WebGPU / Iris Xe 80 EU (8 GB RAM):** peaked at **\~3934 MB — essentially the entire \~3940 MB shared budget** (≈50% of 8 GB RAM). Only 8K ran (11 of 15 runs, \~33 s mean) before it OOM'd; every 4K / 1080p / 512 run failed. WebGPU is **unusable at 8 GB RAM**.
² **DirectML / Iris Xe 96 EU (16 GB RAM):** out of memory (`8007000E`) **during session creation** (\~12.7 s in) — never ran. DirectML's footprint is **\~16–19 GB** (peak where it ran) vs only **\~8032 MB** shared iGPU budget (≈50% of 16 GB RAM).
³ **DirectML / Iris Xe 80 EU (8 GB RAM):** init failed with a GPU **device-hung** error (`887A0007`, "GPU will not respond to more commands") — not strictly OOM, likely a cascade from the preceding WebGPU crash on the same \~3940 MB shared budget.
**CPU backend** (anchor = cores × clock × AVX width + RAM bandwidth; EP = CPU on every row):
| CPU (cores/threads, arch) | RAM BW | 8K (s) | 4K (s) | 1080p (s) | 512² (s) |
|---|--:|--:|--:|--:|--:|
| Core Ultra 7 255H (16C/16T, Arrow Lake-H, AVX2) | \~39 GB/s | 4.57 ± 0.08 | 4.61 ± 0.05 | 4.63 ± 0.12 | 4.65 ± 0.17 |
| i7-8700K (6C/12T, Coffee Lake, AVX2) | \~25 GB/s | 9.75 ± 0.24 | 9.69 ± 0.10 | 9.67 ± 0.11 | 9.67 ± 0.13 |
| i7-1165G7 (4C/8T, Tiger Lake, AVX2/512) | \~31 GB/s | 14.6 ± 1.5 | 13.9 ± 0.5 | 13.8 ± 0.3 | 13.9 ± 0.2 |
| i5-1335U (10C/12T, 15 W, Raptor Lake-U) | \~29 GB/s | 14.3 ± 2.1 | 14.9 ± 1.6 | 15.7 ± 2.0 | 14.3 ± 1.8 |
(The CPU EP was also run on the RTX 5070 Ti box — that's the i7-8700K row — since the CPU EP ignores the GPU. The 1335U's larger ± is real run-to-run jitter on a 15 W thermally-throttled laptop.)
**How to read it / rules of thumb:**
- **Choose the EP, not just the chip.** Discrete NVIDIA/AMD → **WebGPU** (fastest). Intel **Arc** iGPU → **DirectML** (its WebGPU path is \~5× slower here). Older Intel **Iris Xe** → **WebGPU** if you have ≥16 GB RAM, else **CPU**. No usable GPU → **CPU** (scales with cores × clock × AVX width; the 16-core 255H is \~3× the 4-core 1165G7).
- **RAM is the gate for iGPUs.** The fp32 model is \~805 MB on disk but a GPU session peaks at **8–20 GB** (VRAM on discrete cards; *shared system RAM* on iGPUs — WebGPU peaked \~8 GB, DirectML \~15–19 GB in our runs). So: iGPU WebGPU needs **≥16 GB system RAM**; iGPU DirectML wants **≥32 GB**; **8 GB machines can only use the CPU**. Discrete GPUs need **≥8 GB VRAM** free.
- **DirectML is hit-or-miss.** It gave the best iGPU time (Arc 140T, 1.6 s) but failed to initialize on three of four machines (driver/VRAM dependent). Always have the CPU EP as a fallback.
- **Predict GPU time** ≈ scale the RTX 5070 Ti's 0.9 s by its fp16 proxy vs yours, then add \~30–50 % headroom because the model is fp32 and won't fully use tensor cores. (e.g. a \~40 TFLOPS-fp16 card → expect \~2 s; a \~9 TFLOPS iGPU via a *good* runtime → \~1.5–2 s, which matches the Arc 140T on DML.)
## Notes
- **Resolution / batch are fixed** at `1×3×1024×1024` (the network hard-codes 1024² internally). Any aspect ratio works — the squash-resize matches the original model's behavior (a very common approach among mask-generation and image-segmentation models).
- **Runtime requirement:** the graph is opset 17, so it needs **`onnxruntime-node` ≥ 1.12.0** (July 2022; earlier versions will crash on load). Current `onnxruntime-web` is fine.
- **WebGPU:** runs as-is on standard adapters — the widest `Concat`/`Split` is only 4–5 inputs, well under the `maxStorageBuffersPerShaderStage` limit, so no graph surgery is needed (verified on the machines in the Performance section above). Only WebGPU "compatibility mode" (limit 4) devices would need a cascade rewrite.
## License & attribution
**MIT**, inherited from the base model. All credit for the weights and architecture goes to **Finegrain** ([finegrain/finegrain-box-segmenter](https://huggingface.co/finegrain/finegrain-box-segmenter)) and the **MVANet** authors. This repository only provides an ONNX conversion of the public `v0.1` weights (`model.safetensors`, SHA256 `fd5f13919dfc0dda102df1af648c3773c61221aa65fe58d6af978637baded1ae`).
## Citation
```bibtex
@article{mvanet,
title = {Multi-view Aggregation Network for Dichotomous Image Segmentation},
author = {Yu, Qian and Zhao, Xiaoqi and Pang, Youwei and Zhang, Lihe and Lu, Huchuan},
journal = {CVPR},
year = {2024}
}
```
