# Benchmark Datasets

## Evaluation Dataset Requirements

The evaluation set must be separate from any calibration data used for quantization decisions.

### Recommended Composition (minimum 500 samples)

| Category                  | Minimum Samples |
|---------------------------|----------------:|
| English printed documents |              75 |
| Simplified Chinese        |              50 |
| Traditional Chinese       |              50 |
| Chinese-English mixed     |              50 |
| Tables                    |              75 |
| Forms                     |              50 |
| Invoices and receipts     |              50 |
| Mathematical formulas     |              30 |
| Multicolumn documents     |              40 |
| Low-quality scans         |              50 |
| Long text pages           |              30 |

### Suggested Public Sources

- **PubTabNet** — table structure recognition
- **CORD** — consolidated receipt dataset
- **SROIE** — scanned receipts OCR
- **ICDAR 2019** — document images
- **CASIA** — Chinese handwritten/printed
- **Custom Traditional Chinese** — government forms, academic papers

### Directory Structure

```
benchmarks/eval_data/
├── images/           # Input document images (PNG/JPG)
├── ground_truth/     # Reference text files (same stem as image)
└── manifest.json     # Exact sample paths, categories, and source metadata
```

### Calibration vs held-out evaluation (required for schema-3 release)

Sensitivity / LM-head calibration and the final release accuracy run must use
**disjoint** directories. Recommended layout under `eval_dataset/`:

```
eval_dataset/
├── calibration/
│   ├── images/
│   ├── ground_truth/
│   └── manifest.json
└── evaluation/          # held-out; used for release_gate accuracy
    ├── images/
    ├── ground_truth/
    └── manifest.json
```

Build with non-overlapping FUNSD IDs and different synthetic seeds:

```bash
python eval_dataset/build_release_set.py --clean \
  --output-root eval_dataset/calibration \
  --funsd-ids 0,1 --generator-seed 20260720 \
  --funsd-revision <SHA> --latin-font <PATH> --cjk-font <PATH>

python eval_dataset/build_release_set.py --clean \
  --output-root eval_dataset/evaluation \
  --funsd-ids 2,3,4,5 --generator-seed 20260721 \
  --funsd-revision <SHA> --latin-font <PATH> --cjk-font <PATH>
```

`benchmarks/run_accuracy.py` reports aggregate metrics and, when
`manifest.json` lists categories, a `by_category` breakdown. Use
`--save-predictions DIR` to dump hypotheses for failure analysis.

### Ground Truth Format

Each ground truth file is a plain `.txt` file containing the expected OCR output.
For table evaluation, include the expected markdown table structure.

### Evaluation Protocol

1. All models compared using identical:
   - Input images (no preprocessing unless explicitly testing preprocessing)
   - Prompt: `<image>document parsing.`
   - Max tokens: 8192
   - Temperature: 0.0 (deterministic)
   - Output normalization (via `normalize_output.py`)

2. Report per-category results, not just aggregate.

3. Separate raw-input and preprocessed-input results.