Add Phoenix vs Baseer server benchmark and historical evidence

BENCHMARKS.md

@@ -1,5 +1,28 @@
 # Benchmark protocol and claim boundaries
 
+## Additional server comparison: Phoenix vs Baseer__Nakba
+
+These values were produced in an owner-run server experiment and supplied as aggregate results. The model repository does not yet contain the raw line predictions, sample counts, preprocessing manifest, or runnable evaluation bundle for this experiment. They are therefore suitable for publication **with provenance caveats**, but are not labelled independently reproduced or sealed.
+
+| Metric | Phoenix | Baseer__Nakba | Difference |
+|---|---:|---:|---:|
+| Unweighted macro CER over 4 datasets | **9.71%** | 28.67% | −18.97 pp; **66.1% relative reduction** |
+| WER | **34.37%** | 59.52% | −25.15 pp; **42.3% relative reduction** |
+| Exact-line accuracy | 13.38% | **26.38%** | Baseer +13.00 pp |
+| Parameters | **4,988,946** | ≈3.75B | Phoenix ≈752× smaller |
+| File size | **19,936,101 bytes** | ≈7.53 GB | Phoenix ≈378× smaller |
+
+| Dataset | Phoenix CER | Baseer CER | Winner | Phoenix relative CER reduction |
+|---|---:|---:|---|---:|
+| Agapet | **11.13%** | 35.65% | Phoenix | 68.8% |
+| Muharaf | **11.93%** | 25.51% | Phoenix | 53.2% |
+| Omar | 6.88% | **0.48%** | Baseer__Nakba | — |
+| RASAM | **8.90%** | 53.06% | Phoenix | 83.2% |
+
+The macro is an unweighted average over dataset-level CER, not a character-weighted aggregate. Recomputing from the displayed rows gives Phoenix 9.71% exactly and Baseer 28.675%; the source displays 28.67%. WER and exact-line aggregation denominators were not supplied. The result supports a strong efficiency and average-CER claim, but not universal superiority: Baseer wins Omar and exact-line accuracy.
+
+Machine-readable details and publication constraints are in `baseer_server_benchmark.json`.
+
 ## Sealed comparison used for the release decision
 
 - Arms: exp8 and exp9.
@@ -65,6 +88,8 @@ This diagnostic re-evaluates the original Muharaf recognizer, exp6, exp8, and ex
 - exp9 regressed slightly on sealed TariMa.
 - the same-protocol guard table compares the four listed model files fairly on those exact guard inputs.
 - Athar provides system-level functionality that a recognizer weight file alone does not provide.
+- under the supplied server protocol, Phoenix had lower CER on 3/4 datasets, lower unweighted macro CER, and lower WER than Baseer__Nakba while being roughly 752× smaller by parameter count.
+- Baseer__Nakba had lower Omar CER and higher exact-line accuracy in the same supplied comparison.
 
 ## Claims that are not allowed
 
@@ -73,3 +98,29 @@ This diagnostic re-evaluates the original Muharaf recognizer, exp6, exp8, and ex
 - comparing Arabic-normalized CER with another system's raw CER.
 - treating guard-set results as unseen final-test evidence.
 - attributing Athar's retrieval, review, or export capabilities to the exp9 weights.
+- calling the Baseer comparison independently reproduced, sealed, or universally representative until its raw predictions, counts, preprocessing, and evaluation code are archived.
+
+## Historical development results
+
+These rows document the project's progression, but they use **different evaluation sets**. They must not be read as a single head-to-head leaderboard.
+
+| Model | Evaluation set | LM | CER | WER | Character accuracy | Word accuracy |
+|---|---|---:|---:|---:|---:|---:|
+| Muharaf baseline | RASAM-test | No | 37.91% | 88.12% | 62.09% | 11.88% |
+| exp4A | RASAM-test | No | 7.64% | 27.77% | 92.36% | 72.23% |
+| exp6 | RASAM-test | No | 7.48% | 27.17% | 92.52% | 72.83% |
+| exp6 + order-8 LM | RASAM-test | Yes | 6.76% | 27.17% | 93.24% | 72.83% |
+| exp6 + order-8 LM | TariMa-test | Yes | 8.68% | 37.84% | 91.32% | 62.16% |
+| exp7_easy | Easy-Family | No | 5.94% | 22.38% | 94.06% | 77.62% |
+| exp7_hard | Hard-Family | No | 13.70% | 45.61% | 86.30% | 54.39% |
+
+### Historical assistance-layer measurements
+
+| Component | Measurement | Result |
+|---|---|---:|
+| Confidence-gated LM | useful interventions on auto-seg run | 12/18 (67%) |
+| Contextual candidates | difficult lines improved / harmed | 6/10 / 0 |
+| Catalog search | correct work ranked first | 5/5 |
+| Retrieval correction | Qur'an al-ʿAlaq match with reference | 98.3% |
+
+Historical logic-domain LM perplexity was 53.14 for the general model, 40.70 for the self-trained model (−23.4%), and 20.25 for the mixed model (−61.9%). These are language-model diagnostics, not exp9 visual-recognizer CER results.

PUBLISH_CHECKLIST.md

@@ -16,6 +16,8 @@ The repository remains private pending the license review below.
 6. Do not upload the Athar retrieval corpus or raw training datasets with the model.
 7. Keep the wording distinction between sealed release evidence and development-guard diagnostics.
 8. Do not claim comparison with the published Muharaf 18.1% number unless its official protocol is reproduced exactly.
+9. Keep the Phoenix-vs-Baseer section labelled as an owner-run server benchmark until raw predictions, sample counts, preprocessing, and an executable evaluation manifest are archived.
+10. Preserve the counter-results: Baseer__Nakba wins Omar CER and exact-line accuracy; do not replace the scoped conclusion with “Phoenix is universally better.”
 
 ## Suggested commands
 

README.md

@@ -94,6 +94,29 @@ All values below use the same images, references, and greedy decoder for exp8 an
 
 Both Agapet manuscripts improved, and all eleven Omar documents improved. TariMa is the declared exception: exp9 regressed by 0.33 CER percentage points, within the 0.5-point tolerance registered before opening the sealed results.
 
+## Phoenix vs Baseer__Nakba: additional server benchmark
+
+In an additional owner-run server comparison, Phoenix exp9 achieved a substantially lower **four-domain macro CER** and **overall WER** than Baseer__Nakba while using roughly three orders of magnitude fewer parameters. The comparison also exposes an important counter-result: Baseer__Nakba had higher exact-line accuracy and was markedly better on Omar. These aggregate values are reported as supplied; the raw per-line predictions, sample counts, preprocessing manifest, and executable evaluation bundle are not yet included in this repository, so this section is **shareable with that provenance caveat**, not an independently reproduced benchmark.
+
+| Metric | Phoenix exp9 | Baseer__Nakba | Result |
+|---|---:|---:|---|
+| Four-domain unweighted macro CER | **9.71%** | 28.67% | Phoenix: **66.1% relative CER reduction** |
+| Overall WER | **34.37%** | 59.52% | Phoenix: **−25.15 pp**, 42.3% relative reduction |
+| Exact-line accuracy | 13.38% | **26.38%** | Baseer__Nakba higher |
+| Parameters | **4,988,946** | ≈3.75B | Phoenix ≈**752× smaller** |
+| Model file | **19.94 MB** | ≈7.53 GB | Phoenix ≈**378× smaller** |
+
+### Raw CER by dataset
+
+| Dataset | Phoenix exp9 | Baseer__Nakba | Lower CER |
+|---|---:|---:|---|
+| Agapet | **11.13%** | 35.65% | Phoenix (68.8% relative reduction) |
+| Muharaf | **11.93%** | 25.51% | Phoenix (53.2% relative reduction) |
+| Omar | 6.88% | **0.48%** | Baseer__Nakba |
+| RASAM | **8.90%** | 53.06% | Phoenix (83.2% relative reduction) |
+
+The macro CER is an **unweighted mean of the four dataset CER values**, so each dataset contributes equally regardless of line or character count. The four displayed Baseer values average to 28.675%; the supplied 28.67% display is retained, while the unrounded recomputation is recorded in `baseer_server_benchmark.json`. The supported conclusion is therefore: **under this server protocol, Phoenix has lower CER on three of four datasets and a much lower macro CER at a fraction of the model size; Baseer__Nakba remains stronger on Omar and exact-line accuracy.**
+
 ## Same-protocol diagnostic comparison
 
 All models below were decoded on the same pre-cropped line images and raw references with Kraken greedy decoding, without an LM or normalization. Each cell is **CER / word accuracy**.
@@ -198,7 +221,7 @@ These are system capabilities, not properties encoded inside the model weights.
 - Evaluation decoder: greedy, without LM
 - Raw and Arabic-normalized metrics are stored separately; the model card reports raw metrics unless explicitly labelled otherwise.
 
-See `BENCHMARKS.md`, `benchmark_summary.json`, `DATA_AND_LICENSES.md`, and `metadata.json` in this repository for details.
+See `BENCHMARKS.md`, `benchmark_summary.json`, `baseer_server_benchmark.json`, `DATA_AND_LICENSES.md`, and `metadata.json` in this repository for details.
 
 ## Citation
 

SHA256SUMS

@@ -1,3 +1,4 @@
 2896fef9d9665cbb82fba8faa3bf0c628ac6a5cc62eb678f40c707db833aebea  model.mlmodel
-5ec7359346dace3ca43285b5a1e1c9edb0ec0b460092d13e401b70324bfd4750  benchmark_summary.json
-a6b7292250a864d43954a191205e67b2c3a534c488f3bab13a5d9ed620d98a15  metadata.json
+779f134c7ab329431144dfe768065b02a33920eaab639253877da1f12eb80dda  benchmark_summary.json
+3be7fbf28aead97eb65896a5f953c0be356da4c6421a9974a0a71b6c0db261f1  metadata.json
+491f3888e2c2afeb37d2a64f8b8456bdcaa554a8aa29ab1931d86219649304f1  baseer_server_benchmark.json

baseer_server_benchmark.json

@@ -0,0 +1,76 @@
+{
+  "schema_version": 1,
+  "benchmark_name": "Phoenix exp9 vs Baseer__Nakba server comparison",
+  "evidence_status": "owner-supplied aggregate server results; raw predictions, sample counts, and executable evaluation manifest are not included in this model repository",
+  "metric_direction": {
+    "cer": "lower_is_better",
+    "wer": "lower_is_better",
+    "exact_line_accuracy": "higher_is_better"
+  },
+  "overall": {
+    "macro_cer": {
+      "aggregation": "unweighted arithmetic mean across Agapet, Muharaf, Omar, and RASAM",
+      "phoenix_percent": 9.71,
+      "baseer_displayed_percent": 28.67,
+      "baseer_recomputed_percent": 28.675,
+      "absolute_difference_percentage_points": -18.965,
+      "phoenix_relative_error_reduction_percent": 66.13
+    },
+    "wer": {
+      "phoenix_percent": 34.37,
+      "baseer_percent": 59.52,
+      "absolute_difference_percentage_points": -25.15,
+      "phoenix_relative_error_reduction_percent": 42.25,
+      "aggregation_and_denominator": "not supplied"
+    },
+    "exact_line_accuracy": {
+      "phoenix_percent": 13.38,
+      "baseer_percent": 26.38,
+      "winner": "Baseer__Nakba",
+      "aggregation_and_denominator": "not supplied"
+    },
+    "efficiency": {
+      "phoenix_parameters": 4988946,
+      "baseer_parameters_approx": 3750000000,
+      "phoenix_parameter_advantage_approx_x": 751.66,
+      "phoenix_model_bytes": 19936101,
+      "baseer_model_size_approx_gb_decimal": 7.53,
+      "phoenix_file_size_advantage_approx_x": 377.71
+    }
+  },
+  "per_dataset_raw_cer_percent": [
+    {
+      "dataset": "Agapet",
+      "phoenix": 11.13,
+      "baseer": 35.65,
+      "winner": "Phoenix",
+      "phoenix_relative_error_reduction_percent": 68.78
+    },
+    {
+      "dataset": "Muharaf",
+      "phoenix": 11.93,
+      "baseer": 25.51,
+      "winner": "Phoenix",
+      "phoenix_relative_error_reduction_percent": 53.23
+    },
+    {
+      "dataset": "Omar",
+      "phoenix": 6.88,
+      "baseer": 0.48,
+      "winner": "Baseer__Nakba"
+    },
+    {
+      "dataset": "RASAM",
+      "phoenix": 8.90,
+      "baseer": 53.06,
+      "winner": "Phoenix",
+      "phoenix_relative_error_reduction_percent": 83.23
+    }
+  ],
+  "publication_requirements": [
+    "State that Phoenix wins macro CER and three of four dataset CER rows under this server protocol, not that it is universally superior.",
+    "State that Baseer__Nakba wins Omar CER and overall exact-line accuracy.",
+    "Do not describe these aggregates as independently reproduced until raw predictions, sample counts, preprocessing, and evaluation code are archived.",
+    "Do not mix these values with the sealed exp8-vs-exp9 table or the development-guard table; the protocols differ."
+  ]
+}

benchmark_summary.json

@@ -47,6 +47,55 @@
       "relative_character_error_reduction_percent": 25.2719
     }
   },
+  "baseer_server_comparison": {
+    "claim_status": "owner-supplied aggregate server results; share with provenance caveat",
+    "raw_prediction_artifact_included": false,
+    "sample_counts_supplied": false,
+    "overall": {
+      "phoenix_macro_cer": 9.71,
+      "baseer_macro_cer_displayed": 28.67,
+      "baseer_macro_cer_recomputed": 28.675,
+      "phoenix_relative_cer_reduction_percent": 66.13,
+      "phoenix_wer": 34.37,
+      "baseer_wer": 59.52,
+      "wer_difference_percentage_points": -25.15,
+      "phoenix_relative_wer_reduction_percent": 42.25,
+      "phoenix_exact_line_accuracy": 13.38,
+      "baseer_exact_line_accuracy": 26.38
+    },
+    "efficiency": {
+      "phoenix_parameters": 4988946,
+      "baseer_parameters_approx": 3750000000,
+      "phoenix_parameter_advantage_approx_x": 751.66,
+      "phoenix_model_bytes": 19936101,
+      "baseer_model_size_approx_gb_decimal": 7.53,
+      "phoenix_file_size_advantage_approx_x": 377.71
+    },
+    "datasets": [
+      {"name": "Agapet", "phoenix_cer": 11.13, "baseer_cer": 35.65, "winner": "Phoenix"},
+      {"name": "Muharaf", "phoenix_cer": 11.93, "baseer_cer": 25.51, "winner": "Phoenix"},
+      {"name": "Omar", "phoenix_cer": 6.88, "baseer_cer": 0.48, "winner": "Baseer__Nakba"},
+      {"name": "RASAM", "phoenix_cer": 8.90, "baseer_cer": 53.06, "winner": "Phoenix"}
+    ]
+  },
+  "historical_development": {
+    "claim_status": "different evaluation sets; not a single leaderboard",
+    "rows": [
+      {"model": "muharaf_rec_best", "dataset": "RASAM-test", "lm": false, "cer": 37.91, "wer": 88.12, "char_accuracy": 62.09, "word_accuracy": 11.88},
+      {"model": "exp4A", "dataset": "RASAM-test", "lm": false, "cer": 7.64, "wer": 27.77, "char_accuracy": 92.36, "word_accuracy": 72.23},
+      {"model": "exp6", "dataset": "RASAM-test", "lm": false, "cer": 7.48, "wer": 27.17, "char_accuracy": 92.52, "word_accuracy": 72.83},
+      {"model": "exp6 + order-8 LM", "dataset": "RASAM-test", "lm": true, "cer": 6.76, "wer": 27.17, "char_accuracy": 93.24, "word_accuracy": 72.83},
+      {"model": "exp6 + order-8 LM", "dataset": "TariMa-test", "lm": true, "cer": 8.68, "wer": 37.84, "char_accuracy": 91.32, "word_accuracy": 62.16},
+      {"model": "exp7_easy", "dataset": "Easy-Family", "lm": false, "cer": 5.94, "wer": 22.38, "char_accuracy": 94.06, "word_accuracy": 77.62},
+      {"model": "exp7_hard", "dataset": "Hard-Family", "lm": false, "cer": 13.70, "wer": 45.61, "char_accuracy": 86.30, "word_accuracy": 54.39}
+    ],
+    "logic_lm_perplexity": {
+      "general": 53.14,
+      "self_trained": 40.70,
+      "mixed": 20.25,
+      "mixed_relative_reduction_vs_general_percent": 61.9
+    }
+  },
   "same_protocol_development_diagnostic": {
     "claim_status": "development diagnostic; not a sealed final test",
     "protocol": {

metadata.json

@@ -11,7 +11,7 @@
   "python": "3.10.11",
   "pytorch": "2.4.1+cu121",
   "created_date": "2026-08-11",
-  "release_status": "prepared_locally_not_uploaded",
+  "release_status": "uploaded_private_pending_publication_review",
   "license": "CC BY-NC-SA 2.0 (conservative choice due to upstream Muharaf training data)",
   "sealed_release_evidence": {
     "lines": 22442,
@@ -27,5 +27,17 @@
     "lowest_cer_domains": "4/4",
     "highest_word_accuracy_domains": "3/4",
     "claim_status": "development diagnostic, not sealed final test"
+  },
+  "baseer_server_comparison": {
+    "evidence_status": "owner-supplied aggregate server results; reproducibility bundle not included",
+    "datasets": 4,
+    "phoenix_macro_cer": 9.71,
+    "baseer_macro_cer_displayed": 28.67,
+    "phoenix_wer": 34.37,
+    "baseer_wer": 59.52,
+    "phoenix_exact_line_accuracy": 13.38,
+    "baseer_exact_line_accuracy": 26.38,
+    "phoenix_lower_cer_domains": "3/4",
+    "claim_status": "share with provenance caveat"
   }
 }