Donut Base Fine-tuned on CORD v2 Dataset

Model Description

This model is a fine-tuned version of naver-clova-ix/donut-base on the CORD v2 (Consolidated Receipt Dataset) for OCR-free receipt understanding and structured information extraction.

Donut (Document Understanding Transformer) is an end-to-end vision-language model that processes document images directly to extract structured information without requiring an intermediate OCR step. This approach eliminates error propagation from OCR mistakes and provides a more streamlined solution for document understanding tasks.

Model Details

Base Model: naver-clova-ix/donut-base (201M parameters)
Task: Image-to-text generation for structured receipt information extraction
Training Approach: Full fine-tuning (all 201M parameters updated)
Hardware: RTX 5090 GPU with 32GB VRAM
Training Framework: PyTorch Lightning with mixed precision (BF16)
Dataset: CORD v2 - 800 training images, 100 validation images
Image Resolution: 1280×960 pixels
Max Sequence Length: 768 tokens

Performance

Test Set Results (100 held-out receipt images)

Mean Accuracy: 90.73%
Accuracy Range: 27% - 100% across samples
Performance by Complexity:
- Simple receipts (3-5 items): 95-100% accuracy
- Complex receipts (15+ items): 75-80% accuracy

Evaluation Metric

Accuracy is calculated using the JSONParseEvaluator from the Donut framework, which performs field-level comparison of extracted information:

Accuracy = (Number of Correctly Extracted Fields) / (Total Number of Fields)

The evaluator compares predicted and ground truth JSON objects by matching key-value pairs across multiple fields including:

Menu items (name, count, unit price)
Subtotals
Tax amounts
Service charges
Total amounts

Note: The 90.73% accuracy means that on average, approximately 9 out of every 10 information fields are correctly extracted from unseen receipts.

Training Details

Hyperparameters

Parameter	Value
Optimizer	AdamW
Learning Rate	3×10⁻⁵
Weight Decay	0.01
Batch Size	1
Gradient Clipping	1.0
Warmup Steps	300
Max Epochs	100
Early Stopping Patience	3 epochs
Precision	BF16 (mixed precision)

Training Configuration

Validation Frequency: Every 0.2 epochs for fine-grained monitoring
Model Checkpointing: Based on validation accuracy
Optimization Techniques:
- Linear warmup scheduling
- Gradient clipping for stability
- Early stopping to prevent overfitting

Resource Utilization

Peak GPU Memory: ~9GB (28% of available 32GB)
Training Time: ~45 minutes per epoch
Power Consumption: 40-75W during training

Usage

Installation

pip install transformers torch pillow

Inference Example

import re
import json
import torch
from PIL import Image
from transformers import DonutProcessor, VisionEncoderDecoderModel

# Load model and processor
processor = DonutProcessor.from_pretrained("abdirisak874/finetune-donut-cord-v3")
model = VisionEncoderDecoderModel.from_pretrained("abdirisak874/finetune-donut-cord-v3")

device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)

# Load and preprocess image
image = Image.open("path/to/receipt.jpg").convert("RGB")
pixel_values = processor(image, return_tensors="pt").pixel_values
pixel_values = pixel_values.to(device)

# Prepare decoder input
task_prompt = "<s_cord-v2>"
decoder_input_ids = processor.tokenizer(task_prompt, add_special_tokens=False, return_tensors="pt").input_ids
decoder_input_ids = decoder_input_ids.to(device)

# Generate prediction
outputs = model.generate(
    pixel_values,
    decoder_input_ids=decoder_input_ids,
    max_length=model.decoder.config.max_position_embeddings,
    early_stopping=True,
    pad_token_id=processor.tokenizer.pad_token_id,
    eos_token_id=processor.tokenizer.eos_token_id,
    use_cache=True,
    num_beams=1,
    bad_words_ids=[[processor.tokenizer.unk_token_id]],
    return_dict_in_generate=True,
)

# Decode output
sequence = processor.batch_decode(outputs.sequences)[0]
sequence = sequence.replace(processor.tokenizer.eos_token, "").replace(processor.tokenizer.pad_token, "")
sequence = re.sub(r"<.*?>", "", sequence, count=1).strip()  # remove first task start token
result = processor.token2json(sequence)

print(json.dumps(result, indent=2))

Expected Output Format

{
  "menu": [
    {
      "nm": "ITEM NAME",
      "cnt": "1",
      "price": "10.00"
    }
  ],
  "sub_total": {
    "subtotal_price": "10.00"
  },
  "total": {
    "total_price": "11.00",
    "tax_price": "1.00"
  }
}

Limitations and Known Issues

Current Limitations

Complex Layout Handling: Performance degrades on receipts with 15+ line items or dense nested structures
Long Item Names: Tendency to truncate very long menu item names
Character Substitution: Occasional OCR-like errors (e.g., 'O' vs '0', 'l' vs '1')
Price Formatting: Some inconsistencies in decimal point placement
Language Support: Primarily trained on Indonesian restaurant receipts; may require fine-tuning for other languages or receipt formats

Error Distribution

Error Type	Frequency	Impact
Character Substitution	32%	Low
Price Formatting	18%	Medium
Item Name Truncation	25%	High
Token Hallucination	15%	Medium
Missing Items	10%	High

Recommended Use Cases

✅ Good for:

Indonesian restaurant receipts
Receipts with 3-15 line items
Standard receipt layouts
Batch processing of similar receipt formats

⚠️ Use with caution:

Receipts with 15+ items
Non-standard layouts
Heavily damaged or low-quality images
Multi-lingual receipts

Training vs. LoRA Comparison

This model was trained using full fine-tuning (updating all 201M parameters) rather than parameter-efficient methods like LoRA. The comparison:

Approach	Parameters Updated	Accuracy	Training Time
LoRA (rank-8)	4.2M (2.1%)	~13%	18 min/epoch
Full Fine-tuning	201M (100%)	90.73%	45 min/epoch

Full fine-tuning proved essential for document understanding tasks requiring complex visual-linguistic alignment, despite higher computational costs.

Research and Citation

This model was developed as part of research on OCR-free document understanding. For detailed methodology, training procedures, and analysis, please refer to the accompanying research paper.

Key Findings

Document understanding requires extensive parameter updates across both vision and language components
Parameter-efficient methods like LoRA are insufficient for complex multimodal tasks
Proper optimization techniques (warmup, gradient clipping, early stopping) are critical for stability
Mixed precision training (BF16) enables practical implementation with reasonable memory requirements

Future Improvements

Potential areas for enhancement:

Knowledge distillation to create smaller, faster models
Data augmentation for improved robustness
Extension to other document types (invoices, forms, IDs)
Multi-language support
Ensemble methods for challenging cases

Contact and Feedback

For questions, issues, or suggestions for improvement, please open an issue on the model repository or contact the model author.

License

This model is released under the OpenRAIL license, inheriting from the base Donut model license.

Acknowledgments

Base model: naver-clova-ix/donut-base
Dataset: CORD v2
Training infrastructure: vast.ai GPU rental services

Downloads last month: 2

abdirisak874
/

finetune-donut-cord-v3