--- license: openrail datasets: - naver-clova-ix/cord-v2 metrics: - accuracy library_name: transformers pipeline_tag: image-to-text language: - en tags: - donut - document-understanding - ocr-free - receipt-processing - vision-language-model - financial-documents --- # Donut Base Fine-tuned on CORD v2 Dataset ## Model Description This model is a fine-tuned version of [naver-clova-ix/donut-base](https://huggingface.co/naver-clova-ix/donut-base) on the [CORD v2 (Consolidated Receipt Dataset)](https://huggingface.co/datasets/naver-clova-ix/cord-v2) for OCR-free receipt understanding and structured information extraction. **Donut** (Document Understanding Transformer) is an end-to-end vision-language model that processes document images directly to extract structured information without requiring an intermediate OCR step. This approach eliminates error propagation from OCR mistakes and provides a more streamlined solution for document understanding tasks. ## Model Details - **Base Model:** naver-clova-ix/donut-base (201M parameters) - **Task:** Image-to-text generation for structured receipt information extraction - **Training Approach:** Full fine-tuning (all 201M parameters updated) - **Hardware:** RTX 5090 GPU with 32GB VRAM - **Training Framework:** PyTorch Lightning with mixed precision (BF16) - **Dataset:** CORD v2 - 800 training images, 100 validation images - **Image Resolution:** 1280×960 pixels - **Max Sequence Length:** 768 tokens ## Performance ### Test Set Results (100 held-out receipt images) - **Mean Accuracy:** 90.73% - **Accuracy Range:** 27% - 100% across samples - **Performance by Complexity:** - Simple receipts (3-5 items): 95-100% accuracy - Complex receipts (15+ items): 75-80% accuracy ### Evaluation Metric Accuracy is calculated using the JSONParseEvaluator from the Donut framework, which performs field-level comparison of extracted information: ``` Accuracy = (Number of Correctly Extracted Fields) / (Total Number of Fields) ``` The evaluator compares predicted and ground truth JSON objects by matching key-value pairs across multiple fields including: - Menu items (name, count, unit price) - Subtotals - Tax amounts - Service charges - Total amounts **Note:** The 90.73% accuracy means that on average, approximately 9 out of every 10 information fields are correctly extracted from unseen receipts. ## Training Details ### Hyperparameters | Parameter | Value | |-----------|-------| | Optimizer | AdamW | | Learning Rate | 3×10⁻⁵ | | Weight Decay | 0.01 | | Batch Size | 1 | | Gradient Clipping | 1.0 | | Warmup Steps | 300 | | Max Epochs | 100 | | Early Stopping Patience | 3 epochs | | Precision | BF16 (mixed precision) | ### Training Configuration - **Validation Frequency:** Every 0.2 epochs for fine-grained monitoring - **Model Checkpointing:** Based on validation accuracy - **Optimization Techniques:** - Linear warmup scheduling - Gradient clipping for stability - Early stopping to prevent overfitting ### Resource Utilization - **Peak GPU Memory:** ~9GB (28% of available 32GB) - **Training Time:** ~45 minutes per epoch - **Power Consumption:** 40-75W during training ## Usage ### Installation ```bash pip install transformers torch pillow ``` ### Inference Example ```python import re import json import torch from PIL import Image from transformers import DonutProcessor, VisionEncoderDecoderModel # Load model and processor processor = DonutProcessor.from_pretrained("abdirisak874/finetune-donut-cord-v3") model = VisionEncoderDecoderModel.from_pretrained("abdirisak874/finetune-donut-cord-v3") device = "cuda" if torch.cuda.is_available() else "cpu" model.to(device) # Load and preprocess image image = Image.open("path/to/receipt.jpg").convert("RGB") pixel_values = processor(image, return_tensors="pt").pixel_values pixel_values = pixel_values.to(device) # Prepare decoder input task_prompt = "" decoder_input_ids = processor.tokenizer(task_prompt, add_special_tokens=False, return_tensors="pt").input_ids decoder_input_ids = decoder_input_ids.to(device) # Generate prediction outputs = model.generate( pixel_values, decoder_input_ids=decoder_input_ids, max_length=model.decoder.config.max_position_embeddings, early_stopping=True, pad_token_id=processor.tokenizer.pad_token_id, eos_token_id=processor.tokenizer.eos_token_id, use_cache=True, num_beams=1, bad_words_ids=[[processor.tokenizer.unk_token_id]], return_dict_in_generate=True, ) # Decode output sequence = processor.batch_decode(outputs.sequences)[0] sequence = sequence.replace(processor.tokenizer.eos_token, "").replace(processor.tokenizer.pad_token, "") sequence = re.sub(r"<.*?>", "", sequence, count=1).strip() # remove first task start token result = processor.token2json(sequence) print(json.dumps(result, indent=2)) ``` ### Expected Output Format ```json { "menu": [ { "nm": "ITEM NAME", "cnt": "1", "price": "10.00" } ], "sub_total": { "subtotal_price": "10.00" }, "total": { "total_price": "11.00", "tax_price": "1.00" } } ``` ## Limitations and Known Issues ### Current Limitations 1. **Complex Layout Handling:** Performance degrades on receipts with 15+ line items or dense nested structures 2. **Long Item Names:** Tendency to truncate very long menu item names 3. **Character Substitution:** Occasional OCR-like errors (e.g., 'O' vs '0', 'l' vs '1') 4. **Price Formatting:** Some inconsistencies in decimal point placement 5. **Language Support:** Primarily trained on Indonesian restaurant receipts; may require fine-tuning for other languages or receipt formats ### Error Distribution | Error Type | Frequency | Impact | |------------|-----------|--------| | Character Substitution | 32% | Low | | Price Formatting | 18% | Medium | | Item Name Truncation | 25% | High | | Token Hallucination | 15% | Medium | | Missing Items | 10% | High | ### Recommended Use Cases ✅ **Good for:** - Indonesian restaurant receipts - Receipts with 3-15 line items - Standard receipt layouts - Batch processing of similar receipt formats ⚠️ **Use with caution:** - Receipts with 15+ items - Non-standard layouts - Heavily damaged or low-quality images - Multi-lingual receipts ## Training vs. LoRA Comparison This model was trained using **full fine-tuning** (updating all 201M parameters) rather than parameter-efficient methods like LoRA. The comparison: | Approach | Parameters Updated | Accuracy | Training Time | |----------|-------------------|----------|---------------| | LoRA (rank-8) | 4.2M (2.1%) | ~13% | 18 min/epoch | | **Full Fine-tuning** | **201M (100%)** | **90.73%** | 45 min/epoch | Full fine-tuning proved essential for document understanding tasks requiring complex visual-linguistic alignment, despite higher computational costs. ## Research and Citation This model was developed as part of research on OCR-free document understanding. For detailed methodology, training procedures, and analysis, please refer to the accompanying research paper. ### Key Findings 1. Document understanding requires extensive parameter updates across both vision and language components 2. Parameter-efficient methods like LoRA are insufficient for complex multimodal tasks 3. Proper optimization techniques (warmup, gradient clipping, early stopping) are critical for stability 4. Mixed precision training (BF16) enables practical implementation with reasonable memory requirements ## Future Improvements Potential areas for enhancement: - Knowledge distillation to create smaller, faster models - Data augmentation for improved robustness - Extension to other document types (invoices, forms, IDs) - Multi-language support - Ensemble methods for challenging cases ## Contact and Feedback For questions, issues, or suggestions for improvement, please open an issue on the model repository or contact the model author. ## License This model is released under the OpenRAIL license, inheriting from the base Donut model license. ## Acknowledgments - Base model: [naver-clova-ix/donut-base](https://huggingface.co/naver-clova-ix/donut-base) - Dataset: [CORD v2](https://huggingface.co/datasets/naver-clova-ix/cord-v2) - Training infrastructure: vast.ai GPU rental services