rachitpandey26/qwen2.5-vl-3b-idp

Open to contribute?

I'm actively expanding to improve model performance.

If you:

• Have invoice datasets you can legally share
• Want to collaborate on building open IDP datasets
• Want to contribute annotations or synthetic data generation
• Have ideas for improving document extraction benchmarks

Please reach out via:

• Hugging Face Discussions

Qwen2.5-VL-3B — IDP / Invoice Extraction (Full Fine-Tune)

Full-parameter fine-tune of Qwen2.5-VL-3B-Instruct for Intelligent Document Processing (IDP) — structured invoice field extraction. Given a scanned invoice image, it returns a JSON object with fields like vendor, customer, invoice_date, total_amount, currency, and more.

Proof-of-concept / learning project. Trained on 1x H100.

Training

• Base: Qwen/Qwen2.5-VL-3B-Instruct
• Method: full-parameter SFT (ms-swift), 5 epochs, best-checkpoint by val loss
• Data: public web data — small for poc
• Effective batch 16, LR 1e-5, bf16, flash-attention-2

Results (481-sample val set, vs untrained base)

Field	Base model	This model
Valid JSON output	0% (fails)	100%
currency	2.5%	85%
invoice_date	2.9%	74%
total_amount	17.5%	68%
invoice_number	35.7%	57%
vendor.name	0%	48%

Fine-tuning produced large gains on every key field. High accuracy on easy fields (payment_method, email, discount, balance_due: 90-100%). Numeric fields (amounts, invoice numbers) top out ~60-70% — a capacity limit of the 3B base on hard scanned documents. Suitable as a first-pass extractor with human review.

Usage (ms-swift)

from swift.infer_engine import TransformersEngine, InferRequest, RequestConfig
engine = TransformersEngine("rachitpandey26/qwen2.5-vl-3b-idp")
req = InferRequest(messages=[
    {"role": "system", "content": "You are an invoice extraction model. Given a scanned invoice image, extract the structured fields and return ONLY a single JSON object."},
    {"role": "user", "content": "<image>Extract the invoice fields as JSON."},
], images=["invoice.png"])
resp = engine.infer([req], request_config=RequestConfig(max_tokens=512, temperature=0))
print(resp[0].choices[0].message.content)

Limitations

• Numeric fields (invoice numbers, amounts, dates) reach ~60-70% — not reliable enough for fully automated reconciliation without human review.
• Trained on RVL-CDIP-style scanned invoices; may not generalize to very different invoice layouts.
• Inherits the Qwen-Research license restrictions from the base model.

Acknowledgements

Base model by Alibaba Qwen team. Dataset publicaly avaliable on web. Fine-tuned with ms-swift on Modal.