HasanALhomsi/layoutlmv3-cord-invoice

LayoutLMv3 fine-tuned on CORD (Receipt Understanding)

This model is a fine-tuned version of microsoft/layoutlmv3-base on the CORD (Consolidated Receipt Dataset) for document field extraction (token classification).

Instead of just extracting raw text like traditional OCR, this model understands the structure of a receipt/invoice and classifies each word into a specific field (item name, price, quantity, tax, total, etc.), combining text, layout (bounding boxes), and the document image itself.

---

Model Details

• Base model: microsoft/layoutlmv3-base
• Task: Token Classification (NER-style, BIO tagging)
• Training data: CORD — 800 train / 100 dev / 100 test receipt images
• Number of labels: 54 (see full label list below)
• Framework: 🤗 Transformers (LayoutLMv3ForTokenClassification)

---

Intended Uses & Limitations

Intended use

Extracting structured fields from receipt-style documents (item names, prices, quantities, subtotal, tax, total, payment method) as part of a document understanding / data extraction pipeline feeding a database or RAG system.

Limitations

• Trained on only 800 Indonesian retail receipts — may not generalize well to invoices, contracts, forms, or documents in other languages/layouts without additional fine-tuning.
• Requires word-level bounding boxes as input. If your OCR engine returns line-level boxes (e.g. some PaddleOCR versions), you'll need to split lines into individual words with estimated per-word boxes before inference — otherwise BIO tag predictions may be inconsistent.
• OCR quality (blurry scans, unusual fonts) directly impacts extraction accuracy, since this model relies on OCR output as input (it does not read the image text itself).

---

How to Use

from transformers import LayoutLMv3ForTokenClassification, LayoutLMv3Processor
from PIL import Image
import torch

model = LayoutLMv3ForTokenClassification.from_pretrained("your-username/layoutlmv3-cord-invoice")
processor = LayoutLMv3Processor.from_pretrained("your-username/layoutlmv3-cord-invoice", apply_ocr=False)

image = Image.open("receipt.jpg").convert("RGB")

# words and boxes must come from your own OCR step (e.g. PaddleOCR),
# with boxes normalized to a 0-1000 scale
words = [...]          # list[str]
boxes = [...]           # list[[x0, y0, x1, y1]], normalized 0-1000

encoding = processor(
    image, words, boxes=boxes,
    truncation=True, padding="max_length", max_length=512, return_tensors="pt"
)

with torch.no_grad():
    outputs = model(**encoding)

predictions = outputs.logits.argmax(-1).squeeze().tolist()
labels = [model.config.id2label[p] for p in predictions]

⚠️ This model does not include an OCR engine. You must run OCR (e.g. PaddleOCR, Tesseract) first to obtain words and boxes, since apply_ocr=False.

---

Label List

The model uses BIO tagging over the following CORD field categories:

MENU.NM, MENU.CNT, MENU.PRICE, MENU.UNITPRICE, MENU.DISCOUNTPRICE,
MENU.SUB_NM, MENU.SUB_CNT, MENU.SUB_PRICE, MENU.SUB_UNITPRICE, MENU.ETC,
MENU.SUB_ETC, MENU.NUM, MENU.VATYN,
SUB_TOTAL.SUBTOTAL_PRICE, SUB_TOTAL.TAX_PRICE, SUB_TOTAL.SERVICE_PRICE,
SUB_TOTAL.DISCOUNT_PRICE, SUB_TOTAL.ETC, SUB_TOTAL.OTHERSVC_PRICE,
TOTAL.TOTAL_PRICE, TOTAL.CASHPRICE, TOTAL.CHANGEPRICE, TOTAL.CREDITCARDPRICE,
TOTAL.EMONEYPRICE, TOTAL.MENUQTY_CNT, TOTAL.MENUTYPE_CNT, TOTAL.TOTAL_ETC,
VOID_MENU.NM, O

(each prefixed with B- / I- per standard BIO tagging — 54 labels total)

---

Training Procedure

1. Parsed raw CORD JSON annotations (valid_line, dontcare) into word-level (text, bbox, label) triples, preserving original word order within each entity for correct BIO sequencing.
2. Normalized bounding boxes to a 0–1000 scale (clamped to avoid out-of-range values).
3. Tokenized with LayoutLMv3Processor, aligning sub-word tokens to their parent word's label (-100 for non-first sub-tokens, per standard token classification practice).
4. Fine-tuned LayoutLMv3ForTokenClassification using the 🤗 Trainer API.

Training hyperparameters

• Learning rate: 2e-5
• Batch size: 4 (train and eval)
• Epochs: 15
• Evaluation strategy: per epoch

Evaluation

Evaluated using seqeval (entity-level Precision, Recall, F1). See training logs for per-label breakdown.

---

License

The CORD dataset used for fine-tuning is licensed under CC BY 4.0.