paper_v2/PAPER_v2.md

# أثر — An Assistive Hybrid Pipeline for Arabic Manuscript HTR and Scholarly Review

*أثر*

---

## English Abstract

Historical Arabic manuscripts remain largely locked in page images: calligraphic variation, dense
marginalia, page degradation, and domain-specific terminology defeat generic OCR. We present
اثر, an **assistive hybrid pipeline** that couples layout segmentation and Kraken-based
Handwritten Text Recognition (HTR) with six transparent assistance layers: a **confidence-gated

character n-gram language model**, a **layout router** that separates framed logic/philosophy pages
from general pages, a **human-in-the-loop manual/PAGE-XML segmentation** mode, a **contextual LLM

reading assistant**, a **retrieval-augmented corrector** that aligns lines with published digital texts
(Qur'an, ḥadīth), and a **relevance-ranked manuscript catalog**. Building on open resources (Muharaf,
Kraken, eScriptorium), our strongest verified general model with an order-8 language model reaches
**6.76% CER** on RASAM-test and **8.68% CER** on TariMa-test. The gated language model improves
CER on three separated datasets without harming clean pages (e.g., 4.34%→3.29% on the demo set),
and a closed-loop **self-training** experiment reduces logic-domain language-model perplexity by
**62%** (53.14→20.25). The contribution is a reproducible, review-oriented pipeline that keeps the
original transcription intact and surfaces uncertainty rather than silently overwriting it.

**Keywords:** Arabic manuscript HTR; confidence-gated language model; layout routing;
retrieval-augmented correction; human-in-the-loop.

---

## الخلاصة العربية


تبقى المخطوطات العربية التاريخية حبيسة الصور؛ فتنوع الخطوط، وكثافة الحواشي، وتلف الصفحات، والمصطلحات المتخصصة تجعل أنظمة OCR العامة غير كافية. نقدم اثر، وهو خط معالجة هجين مساعد يجمع بين تجزئة التخطيط والتعرف على الخط اليدوي (HTR) بمحرك Kraken، مع ست طبقات مساعدة شفافة: نموذج لغوي حرفي موجه بالثقة، وراوتر تخطيط يميز صفحات المنطق والفلسفة المؤطرة عن الصفحات العامة، ووضع قص يدوي واستيراد PAGE-XML بمشاركة الإنسان، ومساعد قراءة سياقي بنموذج لغوي كبير، ومصحح بالاسترجاع يطابق الأسطر مع نصوص رقمية منشورة، وفهرس مخطوطات مرتب بحسب الصلة. يحقق أقوى نموذج عام متحقق منه معدل خطأ حرفي 6.76% على RASAM-test و 8.68% على TariMa-test عند استخدام نموذج لغوي من الرتبة الثامنة. كما تحسن البوابة اللغوية النتائج على ثلاث مجموعات منفصلة دون الإضرار بالصفحات النظيفة، وتخفض تجربة التدريب الذاتي حيرة النموذج اللغوي في مجال المنطق بنسبة 62%. يهدف النظام إلى تسريع مراجعة المخطوطات مع حفظ النص الأصلي وإظهار مواضع الشك بدلا من استبدالها بصمت.

**الكلمات المفتاحية:** تعرّف الخط اليدوي العربي؛ نموذج لغوي مُبوَّب؛ راوتر تخطيط؛ التصحيح بالاسترجاع؛
مشاركة الإنسان.

---

## 1. Introduction

Digitizing Arabic manuscripts as images does not make them readable, searchable, or analyzable.
Scholarly review needs more than a raw transcript: it needs uncertainty flagging, alternative
readings, term explanation, and cross-referencing with known texts. Generic OCR fails on
handwritten Arabic because a single base shape maps to several letters depending on dots and
scribal convention, and because dense commentary and frames break naïve reading order.

اثر does not start from zero. It builds on open foundations — Muharaf handwritten-Arabic
resources [1], eScriptorium [2], and Kraken [3] — and extends them into an **assistive hybrid

pipeline for recognition *and* review together**. Our contribution is at the engineering and
experimental level: (i) domain fine-tuning that cuts CER from a 37.91% generic baseline to 6.76%;
(ii) a **confidence-gated** language-model decoder that helps hard lines without harming clean ones;
(iii) six transparent assistance layers added around — never replacing — the recognizer; and
(iv) two research probes (retrieval-augmented correction and closed-loop self-training) validated on
real pages.

## 2. Project Problem

Historical Arabic manuscripts exhibit calligraphic variation (Naskh, Maghrebi, Eastern), page
degradation, dense marginalia around framed main text, non-standard orthography, and
domain-specific vocabulary (logic, philosophy, jurisprudence, theology) where one uncertain word can
change an argument. Off-the-shelf recognizers both mis-read these pages and, critically, hide their
uncertainty — unsafe for scholarship.

## 3. Objectives

1. Reduce CER on heterogeneous Arabic manuscripts through data preparation and domain fine-tuning.
2. Add a language layer that helps difficult lines **without degrading easy ones**.
3. Handle complex framed layouts (main text vs. marginalia) via routing and human-in-the-loop tools.
4. Surface uncertainty and provide transparent, non-destructive reading assistance.
5. Keep the whole pipeline reproducible and honest — every claim tied to a measured result.

## 4. Proposed Solution — Hybrid Architecture

The system is a staged pipeline (Figure 1): conservative preprocessing → **layout routing** →
segmentation → HTR recognition → **confidence-gated LM decoding** → a **transparent post-OCR

assistance layer** (candidate readings, retrieval-augmented correction, explanation, catalog search),
plus a **human-in-the-loop** manual/PAGE-XML segmentation path for the hardest pages.

Unlike a plain OCR tool, the assistance layers are *additive and reversible*: the original recognizer
output is preserved, and every suggestion is a labeled annotation the reviewer accepts by a click.

## 5. Innovation and Originality

- **Confidence-gated shallow fusion:** the LM is applied *only* to low-confidence lines, with a
  large-change guard that rejects and flags aggressive rewrites (`uncertain`) — combining the gain of
  language modeling with do-no-harm on clean text.
- **Layout router with frame detection:** a vertical-rule fill test (`cf>0.92`) routes only genuinely
  framed logic pages to the specialized model, fixing a false-positive that had destroyed a normal
  page (58.3%→4.6% CER).
- **Retrieval-augmented correction (RAC):** manuscript lines are aligned to published digital texts;
  Qur'anic/ḥadīth quotations are corrected against the canonical source with a reference.
- **Closed-loop self-training:** the model transcribes more of its own manuscript, high-confidence
  lines become pseudo-labels in the decoder's own encoding, improving the domain LM by 62% perplexity.
- **Human-in-the-loop:** users import eScriptorium PAGE-XML or draw line boxes; column-based
  classification then separates main text from marginalia.

## 6. Methodology

**Preprocessing (safe path).** Recognition models operate on **grayscale (L)** images; binarization
(nlbin), enhancement, resizing, and deskew are **disabled by default**, matching the eScriptorium
setting, because binarizing an L-model input distorts strokes.

**Segmentation & routing.** A fine-tuned baseline segmentation model (`seg_finetuned_v4`, trained on
319 RASAM pages) produces HTR-compatible baselines. A router inspects ruling geometry and density;
only pages with a confirmed rectangular frame use the specialized logic segmentation/recognition
models, otherwise the general models are used.

**Recognition.** Kraken-compatible CTC HTR models (`transcription_finetuned_best` general;
`logic_philosophy_v2_rec` logic). Fine-tuning used a warmup/freeze recipe (`--warmup 2`,
`--freeze-backbone 2`) protecting the visual backbone while sequence layers adapt.

**Gated LM decoding.** An offline character-level 8-gram LM is combined with acoustic confidence
during beam search:

$$\mathrm{Score}(y\mid x) = \log P_{\mathrm{HTR}}(y\mid x) + \alpha \log P_{\mathrm{LM}}(y) + \beta\,|y| \qquad (1)$$

with $\alpha{=}0.5,\ \beta{=}0.3$, beam $=10$. A per-line **confidence gate** (mean character
probability): if confidence $\ge 0.95$ keep greedy; else apply the LM; if the LM changes $>50\%$ of
the line, reject and flag `uncertain`.

**Post-OCR assistance.** Candidate readings (visual beam + contextual LLM), retrieval-augmented
correction, term/page explanation, and catalog search are stored as separate annotations.

**Evaluation.** CER (primary) and WER on frozen splits; the test subset is never used for training
decisions. Three page sets are kept **separate and never mixed**: GT-segmented, manual/GT-demo, and
automatic-segmentation.

## 7. Tools and Technologies

Kraken (HTR engine) · eScriptorium (annotation/PAGE-XML) · PyTorch on **NVIDIA RTX 4050 (CUDA + FP16)** ·
FastAPI backend + PWA frontend · offline character n-gram LM (shallow fusion) · OpenAI GPT-5 family for
the contextual assistant (`gpt-5.4-mini` light tasks, `gpt-5.5` line correction) · OpenITI/Tanzil digital
texts for retrieval. No hardware-speed claims are made in the evidence tables.

## 8. Results / Practical Demonstration

### 8.1 Recognition model comparison (Verified, `metrics_summary.csv`)



The pipeline is a *progression*: a weak generic baseline improved by domain training, mixed-domain

data, and LM rescoring (Table 1, Figure 2).



**Table 1. Model performance (source: `metrics_summary.csv`).**



| Model | Dataset | LM | CER % | WER % | Char Acc % | Word Acc % |

|---|---|---|---:|---:|---:|---:|

| muharaf_rec_best (baseline) | RASAM-test | No | 37.91 | 88.12 | 62.09 | 11.88 |

| exp4A (RASAM-trained) | RASAM-test | No | 7.64 | 27.77 | 92.36 | 72.23 |

| exp6 (RASAM+TariMa general) | RASAM-test | No | 7.48 | 27.17 | 92.52 | 72.83 |

| **exp6 + LM (order-8)** | RASAM-test | Yes | **6.76** | 27.17 | 93.24 | 72.83 |

| exp6 + LM (order-8) | TariMa-test | Yes | 8.68 | 37.84 | 91.32 | 62.16 |

| exp7_easy (specialized) | Easy-Family-test | No | 5.94 | 22.38 | 94.06 | 77.62 |
| exp7_hard (specialized) | Hard-Family-test | No | 13.70 | 45.61 | 86.30 | 54.39 |



Domain fine-tuning cuts CER **37.91%→7.48%**; the order-8 LM adds a further **7.48→6.76%** on

RASAM and **10.27→8.68%** on TariMa (zero-shot domain gap).



### 8.2 Confidence-gated LM — three separated datasets (source: `LM_CONFIDENCE_GATE_REPORT.md`)

**Table 2. Gated LM (threshold <0.95) vs. greedy vs. always-LM (CER %).**

| Dataset | Greedy | Always-LM | **Gated (<0.95)** |
|---|---:|---:|---:|
| D1 clean (GT-seg) | 0.39 | 0.56 | **0.30** |
| D2 demo (GT/manual) | 4.34 | 2.56 | **3.29** |
| D3 auto-seg (v4) | 3.61 | 2.28 | **3.41** |
| Mean (3 sets) | 2.78 | 1.80 | **2.33** |

Always-LM helps hard sets but **harms the clean set** (0.39→0.56). The gate captures most of the gain
while *improving* even the clean set (0.39→0.30) and lowering WER (D2 15.30→10.48). This validates
gating as the safe operating point.

### 8.3 Full-pipeline per-page benchmark (source: `FEATURES_BENCHMARK.md`)



Independent run on 13 held-out pages (automatic segmentation, GPU). Normal pages average **7.63% CER**;

the two 1926-family pages (a known hard automatic-segmentation layout) average 27.30% and drag the

overall mean to 10.65%. During this benchmark we found and fixed a router false-positive (a density

detector had sent normal pages to the logic path, causing a full failure on 1926_0077).

**Table 3. Selected per-page CER (full pipeline, auto-seg).**

| Page | CER % | WER % |
|---|---:|---:|
| 1983_177819 (Eastern) | 1.99 | 7.4 |

| 1977_0136 | 2.92 | 12.7 |
| 417_0011 (after router fix) | 4.61 | 22.5 |

| 1944_0006 | 8.36 | 15.4 |
| 1926_0144 (hard layout) | 29.81 | 54.3 |



Assistance-layer measurements on the same run: gated-LM interventions **12/18 improved** (67% useful,

1 harmful); contextual candidate suggestions **6/10 improved, 0 harmful** on the hardest low-confidence

lines; catalog search returned the correct work first on **5/5** queries after relevance re-ranking.



### 8.4 Self-training the logic LM (source: `RESEARCH_SELF_TRAINING.md`)



Transcribing 60 pages of a logic manuscript (GT pages excluded) yielded **418** high-confidence

(≥0.95) pseudo-labeled lines in the decoder's own visual encoding. A mixed LM (general + pseudo)

reduces perplexity on held-out logic GT by **62%** (Figure 3), confirming that the closed loop models

the domain far better; on the easy GT pages this does not yet move CER (greedy already at 4.21%), but

the mixed LM is consistently the least harmful — the gain is expected on harder/degraded pages.



**Table 4. Logic-domain LM perplexity (lower is better).**



| Language model | Perplexity | vs. general |

|---|---:|---:|

| General (char8_trainval) | 53.14 | — |
| Self-trained (logic only, 418 lines) | 40.70 | −23.4% |
| **Mixed (general + self-trained)** | **20.25** | **−61.9%** |

### 8.5 Retrieval-augmented correction (source: `docs/RETRIEVAL_AUGMENTED_CORRECTION.md`)

Aligning OCR lines to a digital store (Qur'an, Ṣaḥīḥ al-Bukhārī) recovers canonical text with a
reference: e.g., an embedded verse is identified as *Sūrat al-ʿAlaq 1* (98.3% match to the canonical verse),
and prophetic-biography lines match Bukhārī at 95–98%. A calibrated threshold (0.78) cleanly rejects
non-quoting text — no false matches.

## 9. Applications and Impact

The system turns page images into searchable, reviewable text with visible uncertainty: it accelerates
scholarly transcription of logic/philosophy/legal manuscripts, supports catalog-level identification of
works and authors, and — via retrieval — flags and corrects canonical quotations. Because assistance is
additive and reversible, it is safe for critical-edition workflows.

## 10. Challenges and Limitations

- Automatic segmentation still fails on the densest framed layouts (1926 family ≈27% CER); the
  human-in-the-loop PAGE-XML path recovers these (logic pages 32.9%→3.2–3.6% with GT crops).
- Character-level LM rescoring leaves some word-boundary (spacing) errors; measured spacing errors are
  ~16% of all character errors — a target for future word-boundary correction. `Not measured`: isolated
  effect of post-processing normalization; order-10 LM; end-to-end latency benchmark.
- The custom logic subset is 32 pages; inventory reports **1,000** complete lines while the prior paper
  cites **≈1,200** (author audit) — documented rather than silently reconciled.
- Results are validated only for the included datasets/splits; public-data provenance is preserved.

## 11. Conclusion

اثر is a disciplined, reproducible **assistive** HTR pipeline for Arabic manuscripts. The
strongest verified configuration reaches **6.76% CER** (RASAM) / **8.68%** (TariMa). Beyond
recognition, a confidence-gated language model, a layout router, human-in-the-loop segmentation, a
contextual reading assistant, retrieval-augmented correction, and a relevance-ranked catalog form a
transparent review system that preserves the manuscript text and surfaces uncertainty. Two research
probes — retrieval correction and closed-loop self-training (−62% perplexity) — show the architecture
can improve itself without additional hand-labeled data.

## References

[1] M. Saeed, A. Chan, A. Mijar, J. Moukarzel, G. Habchi, C. Younes, A. Elias, C.-W. Wong, A. Khater.
*Muharaf: Manuscripts of Handwritten Arabic Dataset for Cursive Text Recognition.* NeurIPS 2024
Datasets and Benchmarks Track, 2024.
[2] The eScriptorium Project. *eScriptorium: A Digital Text Production Pipeline for Print and

Handwritten Texts using Machine Learning Techniques.* Project documentation.
[3] B. Kiessling. *Kraken: a universal text recognizer for the humanities.* Journal of Open Source
Software, 4(42):1764, 2019. DOI: 10.21105/joss.01764.
[4] A. Graves, S. Fernandez, F. Gomez, and J. Schmidhuber. *Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks.* ICML, 2006.
[5] OpenITI and Tanzil Project. *Open Islamicate Texts Initiative and verified Qur'anic text resources.* Project documentation.

---
*Data ethics: the corpus combines public manuscript resources and project-curated material (≈1,200

author-transcribed lines; automated inventory 1,000). Public-source provenance remains visible; the

cleaning, organization, manual transcription, evaluation, and system integration are the project

contributions. No numbers in this paper are estimated — each is drawn from a committed result file or

marked `Not measured`.*