---
license: cc-by-4.0
base_model: microsoft/layoutlmv3-base
tags:
  - layoutlmv3
  - document-ai
  - token-classification
  - ocr
  - invoice-extraction
  - receipt-extraction
datasets:
  - naver-clova-ix/cord-v2
language:
  - en
pipeline_tag: token-classification
---

# LayoutLMv3 fine-tuned on CORD (Receipt Understanding)

This model is a fine-tuned version of [microsoft/layoutlmv3-base](https://huggingface.co/microsoft/layoutlmv3-base) on the [CORD](https://github.com/clovaai/cord) (Consolidated Receipt Dataset) for document field extraction (token classification).

Instead of just extracting raw text like traditional OCR, this model understands the **structure** of a receipt/invoice and classifies each word into a specific field (item name, price, quantity, tax, total, etc.), combining text, layout (bounding boxes), and the document image itself.

---

## Model Details

- **Base model:** `microsoft/layoutlmv3-base`
- **Task:** Token Classification (NER-style, BIO tagging)
- **Training data:** CORD — 800 train / 100 dev / 100 test receipt images
- **Number of labels:** 54 (see full label list below)
- **Framework:** 🤗 Transformers (`LayoutLMv3ForTokenClassification`)

---

## Intended Uses & Limitations

### Intended use
Extracting structured fields from **receipt-style documents** (item names, prices, quantities, subtotal, tax, total, payment method) as part of a document understanding / data extraction pipeline feeding a database or RAG system.

### Limitations
- Trained on only 800 Indonesian retail receipts — may **not generalize well** to invoices, contracts, forms, or documents in other languages/layouts without additional fine-tuning.
- Requires **word-level** bounding boxes as input. If your OCR engine returns line-level boxes (e.g. some PaddleOCR versions), you'll need to split lines into individual words with estimated per-word boxes before inference — otherwise BIO tag predictions may be inconsistent.
- OCR quality (blurry scans, unusual fonts) directly impacts extraction accuracy, since this model relies on OCR output as input (it does not read the image text itself).

---

## How to Use

```python
from transformers import LayoutLMv3ForTokenClassification, LayoutLMv3Processor
from PIL import Image
import torch

model = LayoutLMv3ForTokenClassification.from_pretrained("your-username/layoutlmv3-cord-invoice")
processor = LayoutLMv3Processor.from_pretrained("your-username/layoutlmv3-cord-invoice", apply_ocr=False)

image = Image.open("receipt.jpg").convert("RGB")

# words and boxes must come from your own OCR step (e.g. PaddleOCR),
# with boxes normalized to a 0-1000 scale
words = [...]          # list[str]
boxes = [...]           # list[[x0, y0, x1, y1]], normalized 0-1000

encoding = processor(
    image, words, boxes=boxes,
    truncation=True, padding="max_length", max_length=512, return_tensors="pt"
)

with torch.no_grad():
    outputs = model(**encoding)

predictions = outputs.logits.argmax(-1).squeeze().tolist()
labels = [model.config.id2label[p] for p in predictions]
```

> ⚠️ This model does **not** include an OCR engine. You must run OCR (e.g. PaddleOCR, Tesseract) first to obtain `words` and `boxes`, since `apply_ocr=False`.

---

## Label List

The model uses BIO tagging over the following CORD field categories:

```
MENU.NM, MENU.CNT, MENU.PRICE, MENU.UNITPRICE, MENU.DISCOUNTPRICE,
MENU.SUB_NM, MENU.SUB_CNT, MENU.SUB_PRICE, MENU.SUB_UNITPRICE, MENU.ETC,
MENU.SUB_ETC, MENU.NUM, MENU.VATYN,
SUB_TOTAL.SUBTOTAL_PRICE, SUB_TOTAL.TAX_PRICE, SUB_TOTAL.SERVICE_PRICE,
SUB_TOTAL.DISCOUNT_PRICE, SUB_TOTAL.ETC, SUB_TOTAL.OTHERSVC_PRICE,
TOTAL.TOTAL_PRICE, TOTAL.CASHPRICE, TOTAL.CHANGEPRICE, TOTAL.CREDITCARDPRICE,
TOTAL.EMONEYPRICE, TOTAL.MENUQTY_CNT, TOTAL.MENUTYPE_CNT, TOTAL.TOTAL_ETC,
VOID_MENU.NM, O
```

(each prefixed with `B-` / `I-` per standard BIO tagging — 54 labels total)

---

## Training Procedure

1. Parsed raw CORD JSON annotations (`valid_line`, `dontcare`) into word-level `(text, bbox, label)` triples, preserving original word order within each entity for correct BIO sequencing.
2. Normalized bounding boxes to a 0–1000 scale (clamped to avoid out-of-range values).
3. Tokenized with `LayoutLMv3Processor`, aligning sub-word tokens to their parent word's label (`-100` for non-first sub-tokens, per standard token classification practice).
4. Fine-tuned `LayoutLMv3ForTokenClassification` using the 🤗 `Trainer` API.

### Training hyperparameters
- Learning rate: `2e-5`
- Batch size: 4 (train and eval)
- Epochs: 15
- Evaluation strategy: per epoch

### Evaluation
Evaluated using `seqeval` (entity-level Precision, Recall, F1). See training logs for per-label breakdown.

---

## License

The CORD dataset used for fine-tuning is licensed under [CC BY 4.0](https://github.com/clovaai/cord/blob/master/LICENSE-CC-BY).