docs/PRD.md

# Product Requirements Document: Verified OCR-Aware Unlimited-OCR for MLX

Status: Approved for implementation  
Owner: AutomatosX  
Target release: `AutomatosX/AX-Unlimited-OCR-3B-MoE-MLX-MXFP8`

## 1. Problem

The current project improves inference ergonomics around an existing third-party
MXFP8 checkpoint, but it does not yet prove that AutomatosX produced distinct,
OCR-aware model weights. The existing precision map has no measured sensitivity
overrides, the evaluation dataset is empty, and the legacy upload path can
republish source weights with only metadata changes.

Users need an Apple Silicon checkpoint whose model-level optimization is
reproducible, measurable, and clearly separated from application tooling.

## 2. Product outcome

Produce and publish a distinct Unlimited-OCR MLX checkpoint converted from the
upstream Baidu BF16 model. Quantization decisions must be derived from OCR
sensitivity measurements, and publication must be blocked unless quality,
performance, provenance, and R-SWA checks pass.

## 3. Users and jobs

- Apple Silicon developers converting PDFs and images to text or Markdown.
- RAG teams requiring local document ingestion with predictable memory use.
- Model engineers who need reproducible quantization and benchmark evidence.
- Reviewers who need to distinguish new weights from repackaged checkpoints.

## 4. Goals

1. Start from an unquantized BF16 `baidu/Unlimited-OCR` checkpoint.
2. Measure BF16 baseline quality before quantization.
3. Measure quantization sensitivity for each supported module group.
4. Keep sensitive groups in BF16, quantize eligible groups to MXFP8, and allow
   a measured 8-bit affine exception when it is the only configuration that
   satisfies both quality and throughput gates.
5. Produce a candidate checkpoint with a deterministic provenance manifest.
6. Compare the candidate with both BF16 and the Sahil MXFP8 reference.
7. Verify native Unlimited-OCR R-SWA behavior with an 8K-or-longer run.
8. Publish atomically to the AutomatosX namespace only after all gates pass.

## 5. Non-goals

- Retraining the Baidu base model or claiming a new foundation model.
- LoRA fine-tuning for a specific document domain.
- Claiming upstream paper scores without reproducing them.
- Publishing an unvalidated checkpoint merely because conversion completed.
- Implementing true multi-image one-shot PDF inference in this release.

## 6. Functional requirements

### FR-1 Source verification

- Reject an already quantized model as the sensitivity or conversion source.
- Record source repository/path, revision when available, config digest, and
  weight-file SHA-256 values.

### FR-2 Evaluation data

- Validate image/ground-truth pairs before allocating the model.
- Record dataset manifest digest and category counts.
- Use disjoint, content-addressed datasets for sensitivity/calibration and the
  final release comparison; shared image hashes block publication.
- Release evaluation must include text, digits, tables, and CJK content.
- Synthetic-only evaluation may be used for development, but not labelled as a
  public accuracy benchmark.

### FR-3 Sensitivity analysis

- Evaluate BF16 once, then quantize one supported group at a time.
- Record matched concrete module paths and CER, digit-CER, and table-score
  deltas for every group.
- Fail if any requested group cannot be evaluated.

### FR-4 Precision-map generation

- Generate executable top-level precision rules; metadata-only nested rules do
  not count.
- Promote a group to BF16 if any configured quality threshold is exceeded.
- Preserve vision, projector, embeddings, normalisation, LM head, and MoE gate
  in BF16 unless explicit evidence supports quantizing them.
- Select any exception through a recorded multi-candidate calibration; a manual
  precision-map edit is not sufficient release evidence.

### FR-5 Conversion

- Convert from BF16 in a staging directory.
- Fail if no module is quantized, a requested rule matches nothing, or no model
  weights are produced.
- Write `precision_map.json` and `quantization_summary.json` beside the weights.
- Record every quantized module's actual mode; the released configuration uses
  MXFP8 for 119 modules and affine INT8 for the LM head.

### FR-6 Comparative validation

- Benchmark BF16, Sahil MXFP8, and the candidate with the same inputs, prompts,
  token budgets, and software versions.
- Store per-sample results as well as aggregates.

### FR-7 Release gates

Publication requires all of the following:

- Candidate weight SHA-256 differs from the Sahil reference.
- Candidate mean CER is no more than 0.01 absolute above BF16.
- Candidate mean CER is no worse than the Sahil reference by more than 0.005.
- Digit CER degradation versus BF16 is no more than 0.01 absolute.
- Table score degradation versus BF16 is no more than 0.01 absolute.
- Candidate decode throughput is at least 90% of the Sahil reference.
- Candidate model weights are no larger than 4.5 GB.
- R-SWA validation reaches at least 8192 generated tokens with bounded-cache
  evidence, or explicitly reports an inconclusive gate and blocks release.
- Every required artifact and digest is present.

### FR-8 Publication

- Create or update `AutomatosX/AX-Unlimited-OCR-3B-MoE-MLX-MXFP8` atomically.
- Upload only the validated candidate weights, metadata, model card, license,
  inference package, benchmark summaries, and release manifest.
- Never silently fall back to uploading Sahil weights.

### FR-9 Claims

- The model card must distinguish upstream paper results, third-party reference
  results, and AutomatosX measurements.
- Failed, skipped, estimated, and inconclusive measurements must be labelled.

## 7. Success metrics

- A public Hugging Face model repository contains distinct candidate weights.
- `release_manifest.json` has `release_approved: true` and every gate result.
- A clean environment can reproduce conversion from the recorded source.
- The model loads through `mlx-vlm` using native `model_type=unlimited-ocr`.
- CLI single-image OCR succeeds with the published model ID.

## 8. Risks

- Evaluation data may be too small or biased to identify fragile layers.
- Group-level sensitivity can hide individual-layer outliers.
- MXFP8 support or module paths may change between `mlx-vlm` versions.
- Long-output documents may terminate naturally before the R-SWA test length.
- Public benchmark licenses may restrict redistribution of evaluation images.

Mitigations include pinned versions, digest-based provenance, fail-closed gates,
per-sample results, and release notes that state dataset limitations.

## 9. Rollout

1. Development: unit tests and synthetic smoke data.
2. Candidate: BF16 sensitivity, conversion, and private/local validation.
3. Release: approve manifest, create public model repository, upload atomically.
4. Post-release: load the exact remote revision and run a final OCR smoke test.