README.md

---


license: mit
base_model: baidu/Unlimited-OCR
library_name: mlx
pipeline_tag: image-text-to-text
tags:
  - ax-engine
  - mlx
  - mlx-vlm
  - ocr
  - mxfp8
  - int8
  - apple-silicon
  - unlimited-ocr
  - automatosx
widget:
- text: "<image>document parsing."
- text: "<image>Convert the document to markdown."
- text: "<image>Free OCR. Extract all text."
---

# AX Unlimited-OCR 3B MoE — MLX (MXFP8 + affine INT8 head)

This is an AutomatosX sensitivity-guided MLX conversion of
[`baidu/Unlimited-OCR`](https://huggingface.co/baidu/Unlimited-OCR). It was
created from the upstream BF16 weights. The
[`sahilchachra/unlimited-ocr-mxfp8-mlx`](https://huggingface.co/sahilchachra/unlimited-ocr-mxfp8-mlx)
checkpoint was used only as a comparison reference; these are not repackaged
Sahil weights.

**Primary inference engine:** [AX Engine](https://github.com/defai-digital/ax-engine)
(`model_family=unlimited_ocr`) — use this for Apple Silicon serving and native
MXFP8 / dual-vision loads.

**Alternative runtime:** `mlx-vlm` (weight-compatible; not the recommended
production path for AutomatosX packs).

**Hybrid precision (name is shorthand):** MXFP8 (group size 32) on eligible
language linears, **affine INT8** on the LM head, **BF16** on the vision
encoder and other protected groups. The Hub “1B params” style size reflects
active MoE capacity; the marketing name refers to the upstream 3B MoE family.

## Release

- **Version tag:** `v0.2.0` (immutable Hub git tag; pin this in production)
- **Hub pin:** `revision="v0.2.0"` or the commit SHA after publish
- **Repo:** [`AutomatosX/AX-Unlimited-OCR-3B-MoE-MLX-MXFP8`](https://huggingface.co/AutomatosX/AX-Unlimited-OCR-3B-MoE-MLX-MXFP8)
- **Runtime:** [AX Engine](https://github.com/defai-digital/ax-engine)
- **Release notes:** Sensitivity-guided MXFP8 with calibrated affine INT8 LM head; primary inference via AX Engine; mlx-vlm alternative only

Immutable upstream source revision: `ee63731b6461c8afcdcc7b15352e7d2ffecc2ead`.  
Reference checkpoint revision: `55b8031a6c867de675279d9604e38cc94b9882a4`.

## What was optimized

Each supported module group was quantized independently and evaluated for OCR
CER, digit CER, and table structure. Eligible language linear layers use MXFP8
(8 data bits, group size 32). A joint calibration then selected affine INT8
(group size 32) for the LM head because BF16 missed the throughput floor and
MXFP8 missed the CER ceiling. OCR-critical or sensitive groups remain BF16.
Retained groups: vision_encoder, vision_projector.

The checkpoint declares `model_type=unlimited-ocr`,
`processor_class=UnlimitedOCRHFProcessor`, and a sliding-window size of
128. Prompts should be raw Unlimited-OCR strings such as
`<image>document parsing.` (not multi-turn chat formatting).

AX Engine maps this pack to native family `unlimited_ocr` (dual vision
SAM-ViT-B + CLIP-L, SWA MoE language tower, MXFP8 dense/expert packs, R-SWA
prefill). Prefer the shipped `model-manifest.json` for AX-ready loads.

## AutomatosX validation

These are **local release measurements**, not upstream paper benchmark claims
and **not** OmniDocBench / olmOCR-Bench scores. Absolute CER on this set is
not a production accuracy bar; gates check **parity with BF16** and packaging
integrity.

The content-addressed held-out evaluation set had 12 samples
(12 with digits, 3 with
CJK, and 3 with tables), digest
`7becb54d81bae0dc69092ff69bae4a324ab63dd672123e5de5d19b6baebbc987`.

| Checkpoint | Mean CER | Digit CER | CJK CER | Table score | Decode tok/s |
|---|---:|---:|---:|---:|---:|
| Upstream BF16 | 0.205212 | 0.113310 | 0.333333 | 1.000000 | — |
| Community MXFP8 reference | 0.907381 | 0.406511 | 1.000000 | 1.000000 | 35.07 |
| AutomatosX candidate | 0.216331 | 0.111939 | 0.341270 | 1.000000 | 35.14 |

**Caveats**
- Small local set; some samples are synthetic. Do not treat mean CER as field accuracy.
- The community reference may use a different `model_type` / load path; it is a
  throughput and packaging baseline, not an identical-recipe peer.
- Prefer **AX Engine** for inference. `mlx-vlm` is an alternative load path only.
- Document helpers (e.g. AX-OCR PDF/profiles) may still call mlx-vlm under the
  hood; pin the same Hub revision when comparing stacks.

### Weight integrity

Weight size: 3.582 GiB  
Aggregate weight SHA-256: `5ff9be1dc0a833b608f6cece5b2e257c5f152eab7de160c5f07d3e5cd9462549`

Per-file weight SHA-256:

- `model.safetensors`: `d9fe11fc8d2e333000be6a0e288681231efa69282c844f7775ca40f52db02f0c`

## Release gates

- ✅ `weights_are_distinct` — actual `5ff9be1dc0a833b608f6cece5b2e257c5f152eab7de160c5f07d3e5cd9462549`, limit `different from 439c6acfe5e277537dfe3368b94a145a6b0da4c39e4bd00582977d030f292ad1`
- ✅ `weight_size_gb` — actual `3.5815872186794877`, limit `4.5`
- ✅ `native_model_metadata` — actual `{'architecture': True, 'model_type': True, 'mxfp8': True, 'sliding_window': True, 'processor_class': True, 'sft_format': True, 'precision_map': True, 'quantization_summary': True, 'protected_modules_preserved': True}`, limit `True`
- ✅ `candidate_source_provenance` — actual `{'model': 'Unlimited-OCR', 'revision': 'ee63731b6461c8afcdcc7b15352e7d2ffecc2ead'}`, limit `{'model': ['baidu/Unlimited-OCR', 'Unlimited-OCR'], 'revision': 'ee63731b6461c8afcdcc7b15352e7d2ffecc2ead'}`
- ✅ `evaluation_coverage` — actual `{'samples': 12, 'digit': 12, 'cjk': 3, 'table': 3}`, limit `all counts > 0`
- ✅ `held_out_evaluation_dataset` — actual `{'passed': True, 'calibration_digest': '40260fc715daa552e7b7dcd75d82441065710eccc8b90892b18826cad6f8dfcb', 'evaluation_digest': '7becb54d81bae0dc69092ff69bae4a324ab63dd672123e5de5d19b6baebbc987', 'overlapping_image_sha256': [], 'overlapping_ground_truth_sha256': []}`, limit `distinct dataset digests and no shared image or ground-truth hashes`
- ✅ `accuracy_aggregates_recomputed` — actual `[True, True, True]`, limit `[True, True, True]`
- ✅ `same_evaluation_samples` — actual `{'counts': [12, 12, 12], 'file_counts': [12, 12, 12]}`, limit `{'count': 12, 'files': ['0001.png', '0002.png', '0003.png', '0004.png', '0005.png', '0006.png', '0007.png', '0008.png', '0009.png', '0010.png', '0011.png', '0012.png']}`
- ✅ `model_identities` — actual `{'bf16': 'Unlimited-OCR', 'reference': 'sahilchachra/unlimited-ocr-mxfp8-mlx', 'candidate': 'AX-Unlimited-OCR-3B-MoE-MLX-MXFP8', 'reference_performance': 'sahilchachra/unlimited-ocr-mxfp8-mlx', 'candidate_performance': 'AX-Unlimited-OCR-3B-MoE-MLX-MXFP8', 'rswa': 'AX-Unlimited-OCR-3B-MoE-MLX-MXFP8'}`, limit `{'bf16': 'Unlimited-OCR', 'reference': 'sahilchachra/unlimited-ocr-mxfp8-mlx', 'candidate': 'AX-Unlimited-OCR-3B-MoE-MLX-MXFP8'}`
- ✅ `immutable_model_revisions` — actual `{'bf16_accuracy': 'ee63731b6461c8afcdcc7b15352e7d2ffecc2ead', 'reference_accuracy': '55b8031a6c867de675279d9604e38cc94b9882a4', 'candidate_accuracy': None, 'reference_performance': '55b8031a6c867de675279d9604e38cc94b9882a4', 'candidate_performance': None}`, limit `{'bf16_accuracy': 'ee63731b6461c8afcdcc7b15352e7d2ffecc2ead', 'reference_accuracy': '55b8031a6c867de675279d9604e38cc94b9882a4', 'candidate_accuracy': None, 'reference_performance': '55b8031a6c867de675279d9604e38cc94b9882a4', 'candidate_performance': None}`
- ✅ `same_accuracy_recipe` — actual `{'prompt': ['<image>document parsing.', '<image>document parsing.', '<image>document parsing.'], 'max_tokens': [1024, 1024, 1024], 'profile': ['accurate', 'accurate', 'accurate'], 'generation_settings': [{'temperature': 0.0, 'top_p': 1.0, 'repetition_penalty': 1.0, 'no_repeat_ngram_size': 35, 'ngram_window': 128}, {'temperature': 0.0, 'top_p': 1.0, 'repetition_penalty': 1.0, 'no_repeat_ngram_size': 35, 'ngram_window': 128}, {'temperature': 0.0, 'top_p': 1.0, 'repetition_penalty': 1.0, 'no_repeat_ngram_size': 35, 'ngram_window': 128}]}`, limit `identical official MLX OCR recipe`
- ✅ `candidate_cer_vs_bf16` — actual `0.011118653228923858`, limit `0.015`
- ✅ `candidate_cer_vs_reference` — actual `-0.691050362380767`, limit `0.005`
- ✅ `candidate_digit_cer_vs_bf16` — actual `-0.0013706317359285375`, limit `0.01`
- ✅ `candidate_table_score_vs_bf16` — actual `0.0`, limit `0.01`
- ✅ `performance_aggregates_recomputed` — actual `{'reference': True, 'candidate': True}`, limit `{'reference': True, 'candidate': True}`
- ✅ `same_performance_setup` — actual `{'image_path': ['test_invoice.png', 'test_invoice.png'], 'prompt': ['<image>document parsing.', '<image>document parsing.'], 'max_tokens': [256, 256], 'num_warmup': [1, 1], 'num_runs': [3, 3], 'system': [{'platform': 'macOS-26.5.2-arm64-arm-64bit-Mach-O', 'processor': 'arm', 'python_version': '3.14.6', 'machine': 'arm64', 'mlx_version': '0.32.0', 'mlx_vlm_version': '0.6.6', 'chip': 'Apple M3 Max', 'total_memory_gb': 128.0}, {'platform': 'macOS-26.5.2-arm64-arm-64bit-Mach-O', 'processor': 'arm', 'python_version': '3.14.6', 'machine': 'arm64', 'mlx_version': '0.32.0', 'mlx_vlm_version': '0.6.6', 'chip': 'Apple M3 Max', 'total_memory_gb': 128.0}]}`, limit `identical setup with at least three complete runs`
- ✅ `candidate_tps_vs_reference` — actual `1.001955642681865`, limit `0.9`
- ✅ `rswa_8k_bounded` — actual `{'pass_conditions': {'cache_bounded': True, 'tps_stable': True, '8k_test_passed': True}, 'tokens': 8192, 'repetition_rate': 0.20496894409937888}`, limit `{'min_tokens': 8192, 'max_repetition_rate': 0.25}`
- ✅ `provenance_matches_release` — actual `{'source': 'baidu/Unlimited-OCR', 'reference': 'sahilchachra/unlimited-ocr-mxfp8-mlx', 'target': 'AutomatosX/AX-Unlimited-OCR-3B-MoE-MLX-MXFP8', 'source_config': {'path': 'config.json', 'sha256': '27246d03fd670904ec9601b1cb0861fbb79ec076830771daa8d943d6229946f9'}, 'source_revision': 'ee63731b6461c8afcdcc7b15352e7d2ffecc2ead', 'reference_revision': '55b8031a6c867de675279d9604e38cc94b9882a4', 'dataset_digest': '7becb54d81bae0dc69092ff69bae4a324ab63dd672123e5de5d19b6baebbc987', 'calibration_dataset_digest': '40260fc715daa552e7b7dcd75d82441065710eccc8b90892b18826cad6f8dfcb'}`, limit `{'source': 'baidu/Unlimited-OCR', 'reference': 'sahilchachra/unlimited-ocr-mxfp8-mlx', 'target': 'AutomatosX/AX-Unlimited-OCR-3B-MoE-MLX-MXFP8', 'source_config_sha256': '27246d03fd670904ec9601b1cb0861fbb79ec076830771daa8d943d6229946f9', 'source_revision': 'ee63731b6461c8afcdcc7b15352e7d2ffecc2ead', 'reference_revision': '55b8031a6c867de675279d9604e38cc94b9882a4', 'dataset_digest': '7becb54d81bae0dc69092ff69bae4a324ab63dd672123e5de5d19b6baebbc987', 'calibration_dataset_digest': '40260fc715daa552e7b7dcd75d82441065710eccc8b90892b18826cad6f8dfcb'}`
- ✅ `calibration_recomputed` — actual `{'selected': {'label': 'affine8-head', 'precision': 'affine8'}, 'models': ['AX-Unlimited-OCR-3B-MoE-MLX-MXFP8-cal-affine8', 'AX-Unlimited-OCR-3B-MoE-MLX-MXFP8-cal-bfloat16', 'AX-Unlimited-OCR-3B-MoE-MLX-MXFP8-cal-mxfp8'], 'dataset_digest': '40260fc715daa552e7b7dcd75d82441065710eccc8b90892b18826cad6f8dfcb', 'input_artifacts': {'calibration_baseline_accuracy': {'filename': 'calibration_baseline_accuracy.json', 'size': 5680, 'sha256': 'ea66cc3190566173dd29c3da3c624b3c950487ebe7b80375d0b159fe46f58cd6'}, 'calibration_reference_performance': {'filename': 'calibration_reference_performance.json', 'size': 2271, 'sha256': '98b8082733af986c054d2d888722bdc28eb92c6accbb3d1d0b70cd2cffc0aae4'}, 'calibration_bfloat16_accuracy': {'filename': 'calibration_bfloat16_accuracy.json', 'size': 5678, 'sha256': '5c124b0ffbc8cda32bc941e54b86d8e83f48f9ab05377e1d63b262909b8b1de0'}, 'calibration_bfloat16_performance': {'filename': 'calibration_bfloat16_performance.json', 'size': 2244, 'sha256': 'dba85444f023def084fcd41fb1a404a18010a003557f85bb2070cfda413ec2b0'}, 'calibration_mxfp8_accuracy': {'filename': 'calibration_mxfp8_accuracy.json', 'size': 5689, 'sha256': '77cbe174170ea41896d26176b30a0320754eef8f62977c08f03f41e161659d26'}, 'calibration_mxfp8_performance': {'filename': 'calibration_mxfp8_performance.json', 'size': 2252, 'sha256': 'e6ad66839447d8b24b7c549fb644a547863a6277ba80d392b9d74572c420099b'}, 'calibration_affine8_accuracy': {'filename': 'calibration_affine8_accuracy.json', 'size': 5657, 'sha256': 'c1f7d2ba7a447c2fb14156e3fabda97b7577035d79bf114ea8ccacec3771ac19'}, 'calibration_affine8_performance': {'filename': 'calibration_affine8_performance.json', 'size': 2247, 'sha256': 'f1c8b7d22cc691797f9e221784f2a747386ea81c5ee1286bcd2860a1670bf89f'}}}`, limit `content-addressed inputs and fastest passing experiment`
- ✅ `sensitivity_matches_calibration_dataset` — actual `{'dataset_digest': '40260fc715daa552e7b7dcd75d82441065710eccc8b90892b18826cad6f8dfcb', 'baseline_metrics_match': True}`, limit `{'dataset_digest': '40260fc715daa552e7b7dcd75d82441065710eccc8b90892b18826cad6f8dfcb', 'baseline_metrics_match': True}`
- ✅ `candidate_precision_map_matches_evidence` — actual `91163d306379c513935c877d53344c628d333cf4eba2f75f0f5f00889affcd75`, limit `91163d306379c513935c877d53344c628d333cf4eba2f75f0f5f00889affcd75`
- ✅ `precision_map_reproducible` — actual `c5f8b260d2e1ff2fc0405c6ef3857829ca85f49b1980a8c599fb03e06788eb1a`, limit `c5f8b260d2e1ff2fc0405c6ef3857829ca85f49b1980a8c599fb03e06788eb1a`

Full per-sample evidence, sensitivity results, precision rules, PRD, ADR, and
technical specification are included under `release/`, `quantization/`, and
`docs/`.

## Usage

Requires **Apple Silicon**.

### Primary — AX Engine

**Use [AX Engine](https://github.com/defai-digital/ax-engine) as the primary inference engine** for this
pack (native `unlimited_ocr`: dual vision + SWA MoE + MXFP8 + R-SWA).

Quantization/release measurements for these weights used: mlx-vlm `0.6.6`, mlx `0.32.0`, python `3.14.6`.

#### Install

```bash
# Homebrew (recommended on macOS)
brew tap defai-digital/ax-engine
brew install defai-digital/ax-engine/ax-engine
ax-engine doctor

# or Python wheel
python3 -m pip install --upgrade "ax-engine[download]>=6.11.0,<7"
```

#### Download and prepare

```bash
# Managed download (writes model-manifest.json when needed)
ax-engine download AutomatosX/AX-Unlimited-OCR-3B-MoE-MLX-MXFP8

# Pin a release tag (recommended for production)
hf download AutomatosX/AX-Unlimited-OCR-3B-MoE-MLX-MXFP8 \
  --revision v0.2.0 \
  --local-dir ./AX-Unlimited-OCR-3B-MoE-MLX-MXFP8

# Ensure AX native manifest is present/valid
ax-engine-bench generate-manifest ./AX-Unlimited-OCR-3B-MoE-MLX-MXFP8 --validate
```

This repository ships `model-manifest.json` so downloads are AX-ready without a
manual convert step when the file is present.

#### Serve

```bash
ax-engine serve ./AX-Unlimited-OCR-3B-MoE-MLX-MXFP8 --port 31418
# or, after managed download of the Hub id:
# ax-engine serve AutomatosX/AX-Unlimited-OCR-3B-MoE-MLX-MXFP8 --download --port 31418
```

OpenAI-compatible routes are exposed on the local server. See the
[AX Engine docs](https://github.com/defai-digital/ax-engine/blob/main/docs/GETTING-STARTED.md) for
install, doctor checks, and API details.

### Alternative — mlx-vlm

`mlx-vlm` can load these weights as a **secondary / alternative** path. Prefer
AX Engine for AutomatosX deployments.

```bash
pip install 'mlx-vlm>=0.6.4'
python -m mlx_vlm.generate \
  --model AutomatosX/AX-Unlimited-OCR-3B-MoE-MLX-MXFP8 \
  --revision v0.2.0 \
  --image document.png \
  --prompt '<image>document parsing.' \
  --max-tokens 4096
```

### Optional — AX-OCR document toolkit

AX-OCR adds deskew, PDF, and generation profiles on top of the same Hub weights.
It is a toolkit, not a replacement for AX Engine as the primary runtime.

```bash
pip install 'ax-ocr[mlx]'
# OCRPipeline / ax-ocr CLI: model_path=AutomatosX/AX-Unlimited-OCR-3B-MoE-MLX-MXFP8, revision=v0.2.0
```

This is quantization/conversion work, not foundation-model retraining or domain
fine-tuning.