license: other
license_name: qwen-research
license_link: https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct/blob/main/LICENSE
base_model: Qwen/Qwen2.5-VL-3B-Instruct
tags:
- invoice-extraction
- document-understanding
- vision-language-model
- qwen2.5-vl
- full-fine-tuning
language:
- en
pipeline_tag: image-text-to-text
Open to contribute?
I'm actively expanding to improve model performance.
If you:
- Have invoice datasets you can legally share
- Want to collaborate on building open IDP datasets
- Want to contribute annotations or synthetic data generation
- Have ideas for improving document extraction benchmarks
Please reach out via:
- Hugging Face Discussions
Qwen2.5-VL-3B — IDP / Invoice Extraction (Full Fine-Tune)
Full-parameter fine-tune of Qwen2.5-VL-3B-Instruct for Intelligent Document Processing (IDP) — structured invoice field extraction. Given a scanned invoice image, it returns a JSON object with fields like vendor, customer, invoice_date, total_amount, currency, and more.
Proof-of-concept / learning project. Trained on 1x H100.
Training
- Base: Qwen/Qwen2.5-VL-3B-Instruct
- Method: full-parameter SFT (ms-swift), 5 epochs, best-checkpoint by val loss
- Data: public web data — small for poc
- Effective batch 16, LR 1e-5, bf16, flash-attention-2
Results (481-sample val set, vs untrained base)
| Field | Base model | This model |
|---|---|---|
| Valid JSON output | 0% (fails) | 100% |
| currency | 2.5% | 85% |
| invoice_date | 2.9% | 74% |
| total_amount | 17.5% | 68% |
| invoice_number | 35.7% | 57% |
| vendor.name | 0% | 48% |
Fine-tuning produced large gains on every key field. High accuracy on easy fields (payment_method, email, discount, balance_due: 90-100%). Numeric fields (amounts, invoice numbers) top out ~60-70% — a capacity limit of the 3B base on hard scanned documents. Suitable as a first-pass extractor with human review.
Usage (ms-swift)
from swift.infer_engine import TransformersEngine, InferRequest, RequestConfig
engine = TransformersEngine("rachitpandey26/qwen2.5-vl-3b-idp")
req = InferRequest(messages=[
{"role": "system", "content": "You are an invoice extraction model. Given a scanned invoice image, extract the structured fields and return ONLY a single JSON object."},
{"role": "user", "content": "<image>Extract the invoice fields as JSON."},
], images=["invoice.png"])
resp = engine.infer([req], request_config=RequestConfig(max_tokens=512, temperature=0))
print(resp[0].choices[0].message.content)
Limitations
- Numeric fields (invoice numbers, amounts, dates) reach ~60-70% — not reliable enough for fully automated reconciliation without human review.
- Trained on RVL-CDIP-style scanned invoices; may not generalize to very different invoice layouts.
- Inherits the Qwen-Research license restrictions from the base model.
Acknowledgements
Base model by Alibaba Qwen team. Dataset publicaly avaliable on web. Fine-tuned with ms-swift on Modal.