Update README.md

README.md

@@ -2,123 +2,196 @@
 license: other
 language:
   - pt
-task_categories:
-  - image-text-to-text
+  - en
+library_name: transformers
+base_model: Nanonets/Nanonets-OCR2-3B
+pipeline_tag: image-text-to-text
 tags:
   - ocr
-  - benchmark
   - document-understanding
+  - structured-extraction
+  - dpo
+  - sft
+  - awq
+  - specialized-small-language-model
   - brazilian-portuguese
-  - text-recognition
-  - handwriting-recognition
-  - legal-documents
-size_categories:
-  - n<1K
+datasets:
+  - dharma-ai/DharmaOCR-Benchmark
 ---
 
-# DharmaOCR-Benchmark
+# DharmaOCR Lite
 
 <p align="center">
   <img src="dharma_logo.png" width="200" alt="Dharma-AI Logo"/>
 </p>
 
-## Overview
+## Introduction
 
-**DharmaOCR-Benchmark** is a 496-instance evaluation suite for OCR models, focused on text extraction in **Brazilian Portuguese**. It covers printed text, handwritten text, and legal/administrative documents — domains that are underrepresented in existing large-scale OCR benchmarks such as OCRBench and olmOCR-Bench.
+**DharmaOCR Lite** is a 3B-parameter Specialized Small Language Model (SSLM) for structured OCR, developed by [Dharma-AI](https://dharma-ai.com). It is designed to extract text from document images into a structured JSON format with explicit `header`, `text`, `footer`, and `margin` fields.
 
-This benchmark was introduced alongside the [DharmaOCR](https://huggingface.co/dharma-ai/DharmaOCR-Lite) family of models. It is designed to evaluate not only transcription quality, but also **text degeneration rate** and **unit inference cost** as first-class metrics.
+DharmaOCR Lite achieves **state-of-the-art performance** on [DharmaOCR-Benchmark](https://huggingface.co/datasets/dharma-ai/DharmaOCR-Benchmark), outperforming all evaluated open-source and commercial baselines — including GPT-4o, GPT-5.4, Claude Opus 4.6, Gemini 3.1 Pro, Google Document AI, Amazon Textract, and olmOCR-2-7B — while being significantly cheaper and faster to run.
 
-## Motivation
+### Key Highlights
 
-Existing OCR benchmarks are not designed to reflect the constraints of Brazilian Portuguese document OCR. In preliminary analyses, performance on general-purpose benchmarks did not reliably predict performance in this setting, where language-specific orthography, domain vocabulary, and document formatting can shift the error profile and amplify text degeneration.
+- **Score: 0.911** on DharmaOCR-Benchmark (highest among all 3B models, surpasses all commercial APIs evaluated)
+- **Text degeneration rate: 0.20%** — the lowest across all models tested
+- **~34% relative cost** compared to olmOCR-2-7B vanilla (reference baseline)
+- **1.464s average time per page** on an NVIDIA L40S GPU
+- Trained with a novel **SFT + DPO** pipeline, where DPO is applied to OCR for the first time to explicitly reduce text degeneration
+- **AWQ-quantized** (FP8) for efficient deployment with negligible quality loss
 
-DharmaOCR-Benchmark addresses this gap by providing a dedicated evaluation protocol for this domain.
+## Model Details
 
-## Dataset Composition
+| Attribute | Value |
+|---|---|
+| **Base model** | [Nanonets-OCR2-3B](https://huggingface.co/Nanonets/Nanonets-OCR2-3B) |
+| **Architecture** | Qwen2.5-VL-3B-Instruct (decoder-only VLM) |
+| **Parameters** | ~3B |
+| **Training pipeline** | SFT → DPO → AWQ Quantization (FP8) |
+| **Output format** | Structured JSON (`header`, `text`, `footer`, `margin`) |
+| **Primary language** | Brazilian Portuguese |
+| **Context length** | 8,192 tokens (generation limit) |
+| **Precision** | FP8 (AWQ) |
+| **License** | Apache 2.0 |
 
-| Subset | Samples | Source | Description |
-|---|---|---|---|
-| **ESTER-Pt** | 363 | [Santos et al., 2023](https://link.springer.com/chapter/10.1007/978-3-031-41679-8_21) | Public benchmark for text recognition in Brazilian Portuguese |
-| **Legal** | 83 | Internal curation | Legal and administrative documents from publicly available sources |
-| **BRESSAY** | 50 | [Neto et al., 2024](https://link.springer.com/chapter/10.1007/978-3-031-70543-4_18) | Handwritten text recognition in Brazilian Portuguese |
-| **Total** | **496** | | |
+## Performance
 
-### Labeling
+### DharmaOCR-Benchmark Results
 
-- **ESTER-Pt** and **BRESSAY**: Labels from their original published benchmarks.
-- **Legal**: Two-stage labeling pipeline inspired by olmOCR-Bench methodology — labels produced by multiple LLMs, followed by a **complete manual audit** of all 83 instances.
+| Model | Score | Degeneration Rate (%) | Time/Page (s) |
+|---|---|---|---|
+| **DharmaOCR Lite (ours)** | **0.911** | **0.20** | **1.464** |
+| DharmaOCR Full (ours, 7B) | 0.925 | 0.40 | 2.132 |
+| olmOCR-2-7B (vanilla) | 0.823 | 1.41 | 4.306 |
+| Nanonets-OCR2-3B (vanilla) | 0.791 | 2.62 | 1.911 |
+| Claude Opus 4.6 | 0.833 | — | — |
+| Gemini 3.1 Pro | 0.820 | — | — |
+| GPT-5.4 | 0.750 | — | — |
+| GPT-4o | 0.635 | — | — |
+| Google Document AI | 0.640 | — | — |
+| Amazon Textract | 0.618 | — | — |
 
-## Evaluation Protocol
+> **DharmaOCR Lite (3B) outperforms olmOCR-2-7B (7B)** — a model with more than twice as many parameters — by ~10% in benchmark score, while being ~3× faster and ~66% cheaper per page.
 
-### Score
+### Benchmark Score Definition
 
-The **DharmaOCR-Benchmark Score** is defined as:
+The DharmaOCR-Benchmark Score is defined as:
 
 ```
 score = (LevenshteinRatio + BLEU) / 2
 ```
 
-| Component | What it captures |
-|---|---|
-| `LevenshteinRatio` | Character-level fidelity (misspellings, missing accents, punctuation errors) |
-| `BLEU` | N-gram sequence preservation (reorderings, dropped spans) |
+where `LevenshteinRatio` captures character-level fidelity and `BLEU` measures n-gram sequence preservation.
 
-### Additional Metrics
+## Quickstart
 
-Beyond transcription quality, the benchmark tracks:
+```python
+from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
+from qwen_vl_utils import process_vision_info
+from PIL import Image
+
+model_name = "dharma-ai/DharmaOCR-Lite"
+
+model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
+    model_name,
+    torch_dtype="auto",
+    device_map="auto",
+)
+processor = AutoProcessor.from_pretrained(model_name)
+
+# Load your document image
+image_path = "document.png"
+
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {"type": "image", "image": image_path},
+            {"type": "text", "text": "Extract the text from this image."},
+        ],
+    }
+]
+
+text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+image_inputs, video_inputs = process_vision_info(messages)
+inputs = processor(
+    text=[text],
+    images=image_inputs,
+    videos=video_inputs,
+    padding=True,
+    return_tensors="pt",
+).to(model.device)
+
+generated_ids = model.generate(**inputs, max_new_tokens=8192)
+generated_ids_trimmed = [
+    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
+]
+output_text = processor.batch_decode(
+    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
+)[0]
+
+print(output_text)
+```
 
-- **Text degeneration rate (%):** Percentage of requests that hit the output-token limit *and* exhibit repeated text spans (detected via n-gram repetition criterion). This is a critical operational metric — degenerate requests inflate cost, increase latency, and reduce throughput for all concurrent requests.
+### Expected Output Format
 
-- **Unit cost per page:** Enables fair comparison between self-hosted models and commercial APIs.
+```json
+{
+    "header": "DOCUMENT TITLE  |  Page 1",
+    "margin": null,
+    "footer": "Journal Name, Vol. X, pp. 1-10, 2025.",
+    "text": "Full main body text extracted from the document..."
+}
+```
 
-## Inference Protocol
+Fields that are not present in the original document are returned as `null`.
 
-To ensure reproducibility, all models are evaluated under a standardized setup:
+## Serving with vLLM
 
-| Parameter | Value |
-|---|---|
-| **GPU** | NVIDIA L40S (48GB GDDR6) |
-| **Instance** | AWS g6e.2xlarge |
-| **Engine** | vLLM |
-| **Max output tokens** | 8,192 |
-| **Temperature** | 0 |
+```bash
+vllm serve dharma-ai/DharmaOCR-Lite \
+    --gpu-memory-utilization 0.90 \
+    --max-model-len 65536 \
+    --max-num-batched-tokens 32000
+```
 
-Cost for local models is estimated from average time per page × hourly infrastructure cost. Cost for commercial APIs uses public pricing (March 2026).
+## Training Details
 
-## Benchmark Results
+### Pipeline
 
-| Model | Score | Degeneration (%) | Type |
-|---|---|---|---|
-| **DharmaOCR Full** (7B) | **0.925** | **0.40** | Open-source (ours) |
-| **DharmaOCR Lite** (3B) | **0.911** | **0.20** | Open-source (ours) |
-| Claude Opus 4.6 | 0.833 | — | Commercial |
-| Qwen3-VL-8B | 0.829 | 5.65 | Open-source |
-| olmOCR-2-7B | 0.823 | 1.41 | Open-source |
-| Gemini 3.1 Pro | 0.820 | — | Commercial |
-| Nanonets-OCR2-3B | 0.791 | 2.62 | Open-source |
-| GPT-5.4 | 0.750 | — | Commercial |
-| Dots OCR | 0.738 | 6.85 | Open-source |
-| GLM-OCR | 0.710 | 11.69 | Open-source |
-| Google Vision | 0.686 | — | Commercial |
-| gemma-3-4b-it | 0.654 | 4.03 | Open-source |
-| Google Document AI | 0.640 | — | Commercial |
-| GPT-4o | 0.635 | — | Commercial |
-| Amazon Textract | 0.618 | — | Commercial |
-| Mistral OCR 3 | 0.574 | — | Commercial |
-
-## Usage
+1. **Supervised Fine-Tuning (SFT):** Full fine-tuning on ~39,680 pages of diverse document types (printed, handwritten, tables, forms, mixed layouts) predominantly in Brazilian Portuguese. Labels were generated by large LLMs (Claude Sonnet 4, Llama 4 Maverick, Gemini 2.5 Pro) and validated by human reviewers.
 
-```python
-from datasets import load_dataset
+2. **Direct Preference Optimization (DPO):** A novel application of DPO to OCR — degenerate model outputs were explicitly used as *rejected* examples to penalize looping/repetitive behavior. Preference pairs were constructed from 5 candidate responses per document, scored by Qwen3-VL-235B as judge, and filtered through a multi-stage policy combining Selective-DPO principles and reward-margin analysis. Final dataset: 49,170 preference pairs.
 
-dataset = load_dataset("dharma-ai/DharmaOCR-Benchmark")
-```
+3. **AWQ Quantization (FP8):** Activation-aware weight quantization preserving visual encoder and output layer in full precision. Reduces cost by ~12% with minimal quality degradation (score: 0.921 → 0.911).
+
+### Infrastructure
+
+- **SFT:** 1× NVIDIA H200, 24 CPU cores, 256 GB RAM
+- **DPO:** 4× NVIDIA H200, 96 CPU cores, 1024 GB RAM
+- **Hyperparameters:** Effective batch size 32, AdamW optimizer, cosine LR schedule
+
+## Why DPO for OCR?
+
+Text degeneration — where models get stuck in repetitive loops — is a critical but underreported problem in OCR systems. It's not just a quality issue: degenerate requests consume disproportionate GPU time, reduce throughput for *all* concurrent requests, and inflate costs.
+
+DharmaOCR Lite applies DPO to reduce degeneration by **87.6%** relative to SFT-only training (from 1.61% → 0.20%), the largest reduction observed across all model families tested.
+
+## DharmaOCR-Benchmark
+
+We release [DharmaOCR-Benchmark](https://huggingface.co/datasets/dharma-ai/DharmaOCR-Benchmark), a 496-instance evaluation suite for OCR in Brazilian Portuguese covering:
+
+- **ESTER-Pt** (363 samples): Printed text recognition
+- **Legal** (83 samples): Legal and administrative documents
+- **BRESSAY** (50 samples): Handwritten text recognition
+
+The benchmark evaluates transcription quality, text degeneration rate, and unit inference cost under a unified protocol.
 
-## Important Notes
+## Limitations
 
-- This benchmark was **not used** for training, model selection, DPO pair construction, or quantization calibration of any DharmaOCR model.
-- The benchmark is primarily designed for **Brazilian Portuguese** document OCR evaluation. Using it to evaluate models on other languages is not recommended.
-- Text degeneration detection uses a combined criterion: (1) the request hits the configured output-token limit, and (2) repeated text spans are detected via n-gram analysis.
+- **Language focus:** Primarily optimized for Brazilian Portuguese documents. Performance on other languages may vary.
+- **Field repetition:** The model may occasionally repeat header/footer content within the `text` field. A post-processing step checking exact matches resolves >80% of occurrences.
+- **Domain scope:** Best results on printed, handwritten, and legal/administrative documents. Highly specialized layouts (e.g., complex engineering drawings) are not covered.
 
 ## Citation