File size: 3,041 Bytes
69e2d5d 7ea68f2 d085421 69e2d5d 0e97454 69e2d5d 217d05d 69e2d5d cac6cae | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 | ---
license: other
license_name: qwen-research
license_link: https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct/blob/main/LICENSE
base_model: Qwen/Qwen2.5-VL-3B-Instruct
tags:
- invoice-extraction
- document-understanding
- vision-language-model
- qwen2.5-vl
- full-fine-tuning
language:
- en
pipeline_tag: image-text-to-text
---
# Open to contribute?
I'm actively expanding to improve model performance.
If you:
- Have invoice datasets you can legally share
- Want to collaborate on building open IDP datasets
- Want to contribute annotations or synthetic data generation
- Have ideas for improving document extraction benchmarks
**Please reach out via:**
- Hugging Face Discussions
# Qwen2.5-VL-3B — IDP / Invoice Extraction (Full Fine-Tune)
Full-parameter fine-tune of **Qwen2.5-VL-3B-Instruct** for **Intelligent Document
Processing (IDP)** — structured invoice field extraction. Given a scanned invoice
image, it returns a JSON object with fields like vendor, customer, invoice_date,
total_amount, currency, and more.
**Proof-of-concept / learning project.** Trained on 1x H100.
## Training
- Base: Qwen/Qwen2.5-VL-3B-Instruct
- Method: full-parameter SFT (ms-swift), 5 epochs, best-checkpoint by val loss
- Data: public web data — small for poc
- Effective batch 16, LR 1e-5, bf16, flash-attention-2
## Results (481-sample val set, vs untrained base)
| Field | Base model | This model |
|---|---|---|
| Valid JSON output | 0% (fails) | **100%** |
| currency | 2.5% | **85%** |
| invoice_date | 2.9% | **74%** |
| total_amount | 17.5% | **68%** |
| invoice_number | 35.7% | **57%** |
| vendor.name | 0% | **48%** |
Fine-tuning produced large gains on every key field. High accuracy on easy
fields (payment_method, email, discount, balance_due: 90-100%). Numeric fields
(amounts, invoice numbers) top out ~60-70% — a capacity limit of the 3B base on
hard scanned documents. Suitable as a first-pass extractor with human review.
## Usage (ms-swift)
```python
from swift.infer_engine import TransformersEngine, InferRequest, RequestConfig
engine = TransformersEngine("rachitpandey26/qwen2.5-vl-3b-idp")
req = InferRequest(messages=[
{"role": "system", "content": "You are an invoice extraction model. Given a scanned invoice image, extract the structured fields and return ONLY a single JSON object."},
{"role": "user", "content": "<image>Extract the invoice fields as JSON."},
], images=["invoice.png"])
resp = engine.infer([req], request_config=RequestConfig(max_tokens=512, temperature=0))
print(resp[0].choices[0].message.content)
```
## Limitations
- Numeric fields (invoice numbers, amounts, dates) reach ~60-70% — not reliable
enough for fully automated reconciliation without human review.
- Trained on RVL-CDIP-style scanned invoices; may not generalize to very
different invoice layouts.
- Inherits the Qwen-Research license restrictions from the base model.
## Acknowledgements
Base model by Alibaba Qwen team. Dataset publicaly avaliable on web. Fine-tuned with ms-swift on Modal.
|