---
license: other
license_name: nutrient-commercial
pipeline_tag: image-to-text
language:
- multilingual
tags:
- document-ai
- form-understanding
- multilingual
- object-detection
- vision-language-model
datasets:
- nutrientdocs/form-field-vlm-benchmark
metrics:
- f1
---
# form-field-vlm ยท _commercial_
**Detect and understand interactive form fields from a page image.** The Nutrient system returns each
field's bounding box, type, text label, and radio-group identity as constrained JSON. The production
system can combine a document-specialized VLM with FF-DETR. The Nutrient VLM performs the semantic work:
field detection, fine field typing, text labeling, and radio-group linking. FF-DETR is an optional
localization component that supplies object boxes only; it does not label fields or link radio buttons.
The runnable weights are commercial and are not downloadable from this repository. This public page is
the product specification and scorecard.
- ๐ฏ **Try it:** [form-field-vlm-demo](https://huggingface.co/spaces/nutrientdocs/form-field-vlm-demo)
- ๐ **Leaderboard:** [form-field-vlm-leaderboard](https://huggingface.co/spaces/nutrientdocs/form-field-vlm-leaderboard)
- ๐ **Benchmark:** [form-field-vlm-benchmark](https://huggingface.co/datasets/nutrientdocs/form-field-vlm-benchmark)
## Headline result
Models are ranked by **Typed F1 at IoU 0.5**. A prediction must overlap the gold widget by at
least 0.5 IoU and return the correct fine field type. This strict threshold rewards usable, tightly
grounded fields; IoU 0.2 is also reported to expose models that find the right area but draw loose boxes.
| System | Typed F1 `.5 / .2` | Box F1 `.5 / .2` | Box Recall `.5 / .2` | sec/page |
|---|---:|---:|---:|---:|
| **Nutrient Hybrid** | **0.477 / 0.516** | **0.594 / 0.671** | **0.790 / 0.892** | 3.24 |
| **Nutrient VLM** | **0.406 / 0.478** | **0.514 / 0.646** | **0.551 / 0.692** | 3.04 |
| **GPT-5.6 Sol** | 0.357 / **0.521** | 0.403 / 0.580 | 0.364 / 0.524 | โ |
The evaluation contains 100 held-out clean form pages and 701 annotated fields. The Nutrient hybrid
achieves **0.477 F1 at IoU 0.5**. The Nutrient VLM alone reaches **0.406**, ahead of the strongest cloud
VLM in this evaluation, **GPT-5.6 Sol** (**0.357**). The hybrid also narrows the strict-to-loose Typed F1 gap from
**0.072** for the VLM alone to **0.039**, evidence that FF-DETR localization converts more approximately
correct detections into strict IoU 0.5 matches.
Cells show `IoU 0.5 / IoU 0.2`; ranking uses only the first Typed F1 value. Typed F1 requires the correct
fine field type. Box F1 measures localization while penalizing missed and extra boxes. Box Recall measures
field coverage without penalizing extra boxes. IoU 0.2 is diagnostic for approximately correct but loose
localization.
## Detailed result
Each detail cell is `Typed F1 @ IoU 0.5 / Typed F1 @ IoU 0.2`. Seconds per page are shown only for models measured in
the same local batch environment; provider batch APIs do not provide comparable request latency.
The benchmark is intentionally small for the first release. Treat it as a transparent, reproducible
comparison set, not a complete estimate of every form distribution. The leaderboard exposes additional
precision, recall, density, return-count, and Box Recall breakdowns.
Cloud systems are labeled with the model generation returned by the evaluated batch: GPT-5.6;
Gemini 3.1 Flash-Lite, Gemini 3.5 Flash, and Gemini 3.1 Pro Preview; and Claude 4.5/4.8/5 as applicable.
Leaderboard details retain both the resolved Gemini `model_version` and the submitted `gemini-*-latest`
alias, so future alias changes cannot alter this benchmark snapshot.
## Output
```json
[
{
"box": [82, 164, 418, 205],
"type": "text",
"label": "Account number",
"group_id": null
}
]
```
Coordinates are `[x0, y0, x1, y1]` on a 0โ1000 page grid. Supported types are `text`,
`choice_checkbox`, `choice_radio`, `choice_select`, and `signature`.
## Intended use and limits
- Empty digital forms are the supported release target.
- Multilingual forms and field labels are supported.
- Very dense pages can require adaptive tiling and longer generation budgets.
## License and attribution
The Nutrient VLM weights are offered under a commercial Nutrient license. The evaluation set is public
and reproducible through
[`nutrientdocs/form-field-vlm-benchmark`](https://huggingface.co/datasets/nutrientdocs/form-field-vlm-benchmark).
The hybrid uses FF-DETR / CommonForms
by jbarrow, Apache-2.0, for object detection and box localization. Benchmark pages derive from CommonForms,
CC-BY-4.0.
> ### ๐ฉ Get access
>
> `form-field-vlm` is commercial and its weights are not downloadable here. To evaluate or deploy it
> on-prem, with documents remaining in your infrastructure, **contact Nutrient:
> [nutrient.io/contact-sales](https://www.nutrient.io/contact-sales/).**
## About the author
This project is maintained and funded by [Nutrient](https://nutrient.io/) - The deterministic document infrastructure enterprises run their highest-stakes workflows on: replayable output, clear exceptions, and full audit trails on the messy, regulated documents where AI alone breaks.