kaan84/sinhala-print-trocr-pilot

TrOCR Sinhala Printed Lines — Pilot Weights

Exploratory pilot run. Not the pre-registered confirmatory experiment. Evaluated only on a held-out split of the same synthetic dataset used for training. No real-document evaluation has been performed. Do not cite these numbers as Sinhala OCR performance.

These are the weights from a single exploratory run completed before the project protocol was pre-registered. They are published for transparency and reproducibility of the pilot, and to make the tokenizer findings below independently verifiable.

The pre-registered protocol, scope, risk register, and release gates live in the main project repository: kaan84/sinhala-print-trocr.

Status

Item	Status
Run type	Exploratory pilot (1 seed)
Confirmatory runs (3 seeds, pre-registered config)	Not started
Real-document evaluation	Not started
Dataset license (sinhala_synthetic_ocr-large)	MIT, confirmed
Checkpoint license (Ransaka/TrOCR-Sinhala)	MIT, confirmed
Public redistribution of weights	MIT, confirmed

Relationship to the pre-registered protocol

The pilot used a configuration that differs from the pre-registered confirmatory experiment. Both are reported to keep the record honest:

Parameter	Pilot (this model)	Pre-registered confirmatory
Learning rate	5e-5	4e-5
Epochs	8	10
Split	80/10/10 synthetic	90/10 synthetic train/val; real test
Runs	1	3 seeds, mean ± SD
Test set	Synthetic held-out	Real line-level benchmark

The pilot's purpose was to establish that the pipeline runs end to end and to surface data and tokenizer issues. It succeeded at both: the tokenizer audit below directly shaped the pre-registered protocol. Pilot and confirmatory results are not comparable and will never be pooled.

Pilot results (synthetic only)

Test set: 697 held-out samples from Ransaka/sinhala_synthetic_ocr-large (80/10/10 split, seed 42) — the same generator, fonts, and text distribution as the training data. Metrics use NFC normalization and whitespace collapsing, with predictions and references decoded through the same tokenizer.

Metric	Zero-shot Ransaka/TrOCR-Sinhala	Pilot fine-tune
CER ↓	36.18%	16.98%
WER ↓	55.93%	32.11%
Exact match ↑	0.00%	3.87%

Relative CER reduction: 53.1% under identical evaluation. Validation CER was still falling at the final epoch (17.72%), indicating the run was data/training-limited rather than overfitted.

model-index metadata is deliberately omitted: per the project's release gates, structured metrics are added only once real-document results exist, so synthetic figures are not mistaken for benchmark results.

Context, not comparison: published page-level results on real Sinhala documents (Tesseract v5 10.69% CER; Surya 8.84% CER / 26.64% WER; Google Document AI 2.06% CER; fine-tuned LightOnOCR-2-1B 1.05% CER / 5.63% WER, arXiv:2606.29378) use a different evaluation unit and real pages. They are not directly comparable to the synthetic line-level figures above.

Tokenizer audit (key pilot finding)

Run on 5,574 training texts before training:

Diagnostic	Result	Verdict
Unknown-token rate	0.0000%	Pass
Tokens per grapheme cluster (mean)	0.301	Pass
Encode/decode round-trip failure	30.14% (1,680 texts)	Fail
— spacing-only differences	1,313 (23.6%)	Metric-neutral here
— residual character loss	367 (6.6%)	Hard ceiling
— residual involving ZWJ/ZWNJ	357 (6.4%)	Hard ceiling

Most-lost characters: U+200D ZERO WIDTH JOINER (×309), U+0020 SPACE (×293), U+200C ZERO WIDTH NON-JOINER (×180).

Interpretation. Vocabulary coverage is fine (no unknown tokens), but the inherited tokenizer cannot represent ZWJ/ZWNJ, so Sinhala conjunct forms that require them (e.g. ශ්‍රී, මාධ්‍ය) can never be produced correctly regardless of training — a hard ceiling on roughly 6.6% of texts. Separately, original punctuation spacing is destroyed and re-inserted on decode; this cancels out in the metrics above because predictions and references pass through the same tokenizer, but it will not cancel when comparing against external systems on raw ground truth. Anyone building on these weights inherits both issues. A scoped fix (vocabulary addition plus decoder embedding resize) is planned.

Training configuration

Parameter	Value
Base checkpoint	Ransaka/TrOCR-Sinhala (315.22M params)
Epochs / LR	8 / 5e-5, linear, 10% warmup
Effective batch	8 (2 × grad-accum 4)
Weight decay / precision	0.01 / FP16
Max target length	128 tokens (99th-percentile rule, clamped to [128, 256])
Decoding	beam 4, no-repeat trigram
Checkpoint selection	lowest validation CER (epoch 8)
Seed / hardware	42 / 1× NVIDIA T4 (Colab), ~1h55m

Preprocessing: NFC normalization and whitespace collapsing of labels; padding masked to −100. One record with null text was removed before splitting (6,969 → 6,968). TensorBoard logs are in runs/.

Usage

from PIL import Image
from transformers import TrOCRProcessor, VisionEncoderDecoderModel

repo_id = "kaan84/sinhala-print-trocr-pilot"
processor = TrOCRProcessor.from_pretrained(repo_id)
model = VisionEncoderDecoderModel.from_pretrained(repo_id)

image = Image.open("sinhala-line.png").convert("RGB")
pixel_values = processor(images=image, return_tensors="pt").pixel_values
generated_ids = model.generate(
    pixel_values,
    num_beams=4,
    no_repeat_ngram_size=3,
    max_length=128
)
print(processor.batch_decode(generated_ids, skip_special_tokens=True)[0])