--- license: mit base_model: baidu/Unlimited-OCR library_name: mlx pipeline_tag: image-text-to-text tags: - ax-engine - mlx - mlx-vlm - ocr - mxfp8 - int8 - apple-silicon - unlimited-ocr widget: - text: "document parsing." - text: "Convert the document to markdown." - text: "Free OCR. Extract all text." --- # AX Unlimited-OCR 3B MoE — MLX (MXFP8 + affine INT8 head) This is an AutomatosX sensitivity-guided MLX conversion of [`baidu/Unlimited-OCR`](https://huggingface.co/baidu/Unlimited-OCR). It was created from the upstream BF16 weights. The [`sahilchachra/unlimited-ocr-mxfp8-mlx`](https://huggingface.co/sahilchachra/unlimited-ocr-mxfp8-mlx) checkpoint was used only as a comparison reference; these are not repackaged Sahil weights. **Primary inference engine:** [AX Engine](https://github.com/defai-digital/ax-engine) (`model_family=unlimited_ocr`) — use this for Apple Silicon serving and native MXFP8 / dual-vision loads. **Alternative runtime:** `mlx-vlm` (weight-compatible; not the recommended production path for AutomatosX packs). **Hybrid precision (name is shorthand):** MXFP8 (group size 32) on eligible language linears, **affine INT8** on the LM head, **BF16** on the vision encoder and other protected groups. The Hub “1B params” style size reflects active MoE capacity; the marketing name refers to the upstream 3B MoE family. ## Release - **Version tag:** `v0.2.0` (immutable Hub git tag; pin this in production) - **Hub pin:** `revision="v0.2.0"` or the commit SHA after publish - **Repo:** [`AutomatosX/AX-Unlimited-OCR-3B-MoE-MLX-MXFP8`](https://huggingface.co/AutomatosX/AX-Unlimited-OCR-3B-MoE-MLX-MXFP8) - **Runtime:** [AX Engine](https://github.com/defai-digital/ax-engine) - **Release notes:** Sensitivity-guided MXFP8 with calibrated affine INT8 LM head; primary inference via AX Engine; mlx-vlm alternative only Immutable upstream source revision: `ee63731b6461c8afcdcc7b15352e7d2ffecc2ead`. Reference checkpoint revision: `55b8031a6c867de675279d9604e38cc94b9882a4`. ## What was optimized Each supported module group was quantized independently and evaluated for OCR CER, digit CER, and table structure. Eligible language linear layers use MXFP8 (8 data bits, group size 32). A joint calibration then selected affine INT8 (group size 32) for the LM head because BF16 missed the throughput floor and MXFP8 missed the CER ceiling. OCR-critical or sensitive groups remain BF16. Retained groups: vision_encoder, vision_projector. The checkpoint declares `model_type=unlimited-ocr`, `processor_class=UnlimitedOCRHFProcessor`, and a sliding-window size of 128. Prompts should be raw Unlimited-OCR strings such as `document parsing.` (not multi-turn chat formatting). AX Engine maps this pack to native family `unlimited_ocr` (dual vision SAM-ViT-B + CLIP-L, SWA MoE language tower, MXFP8 dense/expert packs, R-SWA prefill). Prefer the shipped `model-manifest.json` for AX-ready loads. ## AutomatosX validation These are **local release measurements**, not upstream paper benchmark claims and **not** OmniDocBench / olmOCR-Bench scores. Absolute CER on this set is not a production accuracy bar; gates check **parity with BF16** and packaging integrity. The content-addressed held-out evaluation set had 12 samples (12 with digits, 3 with CJK, and 3 with tables), digest `7becb54d81bae0dc69092ff69bae4a324ab63dd672123e5de5d19b6baebbc987`. | Checkpoint | Mean CER | Digit CER | CJK CER | Table score | Decode tok/s | |---|---:|---:|---:|---:|---:| | Upstream BF16 | 0.205212 | 0.113310 | 0.333333 | 1.000000 | — | | Community MXFP8 reference | 0.907381 | 0.406511 | 1.000000 | 1.000000 | 35.07 | | AutomatosX candidate | 0.216331 | 0.111939 | 0.341270 | 1.000000 | 35.14 | **Caveats** - Small local set; some samples are synthetic. Do not treat mean CER as field accuracy. - The community reference may use a different `model_type` / load path; it is a throughput and packaging baseline, not an identical-recipe peer. - Prefer **AX Engine** for inference. `mlx-vlm` is an alternative load path only. - Document helpers (e.g. AX-OCR PDF/profiles) may still call mlx-vlm under the hood; pin the same Hub revision when comparing stacks. ### Weight integrity Weight size: 3.582 GiB Aggregate weight SHA-256: `5ff9be1dc0a833b608f6cece5b2e257c5f152eab7de160c5f07d3e5cd9462549` Per-file weight SHA-256: - `model.safetensors`: `d9fe11fc8d2e333000be6a0e288681231efa69282c844f7775ca40f52db02f0c` ## Release gates - ✅ `weights_are_distinct` — actual `5ff9be1dc0a833b608f6cece5b2e257c5f152eab7de160c5f07d3e5cd9462549`, limit `different from 439c6acfe5e277537dfe3368b94a145a6b0da4c39e4bd00582977d030f292ad1` - ✅ `weight_size_gb` — actual `3.5815872186794877`, limit `4.5` - ✅ `native_model_metadata` — actual `{'architecture': True, 'model_type': True, 'mxfp8': True, 'sliding_window': True, 'processor_class': True, 'sft_format': True, 'precision_map': True, 'quantization_summary': True, 'protected_modules_preserved': True}`, limit `True` - ✅ `candidate_source_provenance` — actual `{'model': 'Unlimited-OCR', 'revision': 'ee63731b6461c8afcdcc7b15352e7d2ffecc2ead'}`, limit `{'model': ['baidu/Unlimited-OCR', 'Unlimited-OCR'], 'revision': 'ee63731b6461c8afcdcc7b15352e7d2ffecc2ead'}` - ✅ `evaluation_coverage` — actual `{'samples': 12, 'digit': 12, 'cjk': 3, 'table': 3}`, limit `all counts > 0` - ✅ `held_out_evaluation_dataset` — actual `{'passed': True, 'calibration_digest': '40260fc715daa552e7b7dcd75d82441065710eccc8b90892b18826cad6f8dfcb', 'evaluation_digest': '7becb54d81bae0dc69092ff69bae4a324ab63dd672123e5de5d19b6baebbc987', 'overlapping_image_sha256': [], 'overlapping_ground_truth_sha256': []}`, limit `distinct dataset digests and no shared image or ground-truth hashes` - ✅ `accuracy_aggregates_recomputed` — actual `[True, True, True]`, limit `[True, True, True]` - ✅ `same_evaluation_samples` — actual `{'counts': [12, 12, 12], 'file_counts': [12, 12, 12]}`, limit `{'count': 12, 'files': ['0001.png', '0002.png', '0003.png', '0004.png', '0005.png', '0006.png', '0007.png', '0008.png', '0009.png', '0010.png', '0011.png', '0012.png']}` - ✅ `model_identities` — actual `{'bf16': 'Unlimited-OCR', 'reference': 'sahilchachra/unlimited-ocr-mxfp8-mlx', 'candidate': 'AX-Unlimited-OCR-3B-MoE-MLX-MXFP8', 'reference_performance': 'sahilchachra/unlimited-ocr-mxfp8-mlx', 'candidate_performance': 'AX-Unlimited-OCR-3B-MoE-MLX-MXFP8', 'rswa': 'AX-Unlimited-OCR-3B-MoE-MLX-MXFP8'}`, limit `{'bf16': 'Unlimited-OCR', 'reference': 'sahilchachra/unlimited-ocr-mxfp8-mlx', 'candidate': 'AX-Unlimited-OCR-3B-MoE-MLX-MXFP8'}` - ✅ `immutable_model_revisions` — actual `{'bf16_accuracy': 'ee63731b6461c8afcdcc7b15352e7d2ffecc2ead', 'reference_accuracy': '55b8031a6c867de675279d9604e38cc94b9882a4', 'candidate_accuracy': None, 'reference_performance': '55b8031a6c867de675279d9604e38cc94b9882a4', 'candidate_performance': None}`, limit `{'bf16_accuracy': 'ee63731b6461c8afcdcc7b15352e7d2ffecc2ead', 'reference_accuracy': '55b8031a6c867de675279d9604e38cc94b9882a4', 'candidate_accuracy': None, 'reference_performance': '55b8031a6c867de675279d9604e38cc94b9882a4', 'candidate_performance': None}` - ✅ `same_accuracy_recipe` — actual `{'prompt': ['document parsing.', 'document parsing.', 'document parsing.'], 'max_tokens': [1024, 1024, 1024], 'profile': ['accurate', 'accurate', 'accurate'], 'generation_settings': [{'temperature': 0.0, 'top_p': 1.0, 'repetition_penalty': 1.0, 'no_repeat_ngram_size': 35, 'ngram_window': 128}, {'temperature': 0.0, 'top_p': 1.0, 'repetition_penalty': 1.0, 'no_repeat_ngram_size': 35, 'ngram_window': 128}, {'temperature': 0.0, 'top_p': 1.0, 'repetition_penalty': 1.0, 'no_repeat_ngram_size': 35, 'ngram_window': 128}]}`, limit `identical official MLX OCR recipe` - ✅ `candidate_cer_vs_bf16` — actual `0.011118653228923858`, limit `0.015` - ✅ `candidate_cer_vs_reference` — actual `-0.691050362380767`, limit `0.005` - ✅ `candidate_digit_cer_vs_bf16` — actual `-0.0013706317359285375`, limit `0.01` - ✅ `candidate_table_score_vs_bf16` — actual `0.0`, limit `0.01` - ✅ `performance_aggregates_recomputed` — actual `{'reference': True, 'candidate': True}`, limit `{'reference': True, 'candidate': True}` - ✅ `same_performance_setup` — actual `{'image_path': ['test_invoice.png', 'test_invoice.png'], 'prompt': ['document parsing.', 'document parsing.'], 'max_tokens': [256, 256], 'num_warmup': [1, 1], 'num_runs': [3, 3], 'system': [{'platform': 'macOS-26.5.2-arm64-arm-64bit-Mach-O', 'processor': 'arm', 'python_version': '3.14.6', 'machine': 'arm64', 'mlx_version': '0.32.0', 'mlx_vlm_version': '0.6.6', 'chip': 'Apple M3 Max', 'total_memory_gb': 128.0}, {'platform': 'macOS-26.5.2-arm64-arm-64bit-Mach-O', 'processor': 'arm', 'python_version': '3.14.6', 'machine': 'arm64', 'mlx_version': '0.32.0', 'mlx_vlm_version': '0.6.6', 'chip': 'Apple M3 Max', 'total_memory_gb': 128.0}]}`, limit `identical setup with at least three complete runs` - ✅ `candidate_tps_vs_reference` — actual `1.001955642681865`, limit `0.9` - ✅ `rswa_8k_bounded` — actual `{'pass_conditions': {'cache_bounded': True, 'tps_stable': True, '8k_test_passed': True}, 'tokens': 8192, 'repetition_rate': 0.20496894409937888}`, limit `{'min_tokens': 8192, 'max_repetition_rate': 0.25}` - ✅ `provenance_matches_release` — actual `{'source': 'baidu/Unlimited-OCR', 'reference': 'sahilchachra/unlimited-ocr-mxfp8-mlx', 'target': 'AutomatosX/AX-Unlimited-OCR-3B-MoE-MLX-MXFP8', 'source_config': {'path': 'config.json', 'sha256': '27246d03fd670904ec9601b1cb0861fbb79ec076830771daa8d943d6229946f9'}, 'source_revision': 'ee63731b6461c8afcdcc7b15352e7d2ffecc2ead', 'reference_revision': '55b8031a6c867de675279d9604e38cc94b9882a4', 'dataset_digest': '7becb54d81bae0dc69092ff69bae4a324ab63dd672123e5de5d19b6baebbc987', 'calibration_dataset_digest': '40260fc715daa552e7b7dcd75d82441065710eccc8b90892b18826cad6f8dfcb'}`, limit `{'source': 'baidu/Unlimited-OCR', 'reference': 'sahilchachra/unlimited-ocr-mxfp8-mlx', 'target': 'AutomatosX/AX-Unlimited-OCR-3B-MoE-MLX-MXFP8', 'source_config_sha256': '27246d03fd670904ec9601b1cb0861fbb79ec076830771daa8d943d6229946f9', 'source_revision': 'ee63731b6461c8afcdcc7b15352e7d2ffecc2ead', 'reference_revision': '55b8031a6c867de675279d9604e38cc94b9882a4', 'dataset_digest': '7becb54d81bae0dc69092ff69bae4a324ab63dd672123e5de5d19b6baebbc987', 'calibration_dataset_digest': '40260fc715daa552e7b7dcd75d82441065710eccc8b90892b18826cad6f8dfcb'}` - ✅ `calibration_recomputed` — actual `{'selected': {'label': 'affine8-head', 'precision': 'affine8'}, 'models': ['AX-Unlimited-OCR-3B-MoE-MLX-MXFP8-cal-affine8', 'AX-Unlimited-OCR-3B-MoE-MLX-MXFP8-cal-bfloat16', 'AX-Unlimited-OCR-3B-MoE-MLX-MXFP8-cal-mxfp8'], 'dataset_digest': '40260fc715daa552e7b7dcd75d82441065710eccc8b90892b18826cad6f8dfcb', 'input_artifacts': {'calibration_baseline_accuracy': {'filename': 'calibration_baseline_accuracy.json', 'size': 5680, 'sha256': 'ea66cc3190566173dd29c3da3c624b3c950487ebe7b80375d0b159fe46f58cd6'}, 'calibration_reference_performance': {'filename': 'calibration_reference_performance.json', 'size': 2271, 'sha256': '98b8082733af986c054d2d888722bdc28eb92c6accbb3d1d0b70cd2cffc0aae4'}, 'calibration_bfloat16_accuracy': {'filename': 'calibration_bfloat16_accuracy.json', 'size': 5678, 'sha256': '5c124b0ffbc8cda32bc941e54b86d8e83f48f9ab05377e1d63b262909b8b1de0'}, 'calibration_bfloat16_performance': {'filename': 'calibration_bfloat16_performance.json', 'size': 2244, 'sha256': 'dba85444f023def084fcd41fb1a404a18010a003557f85bb2070cfda413ec2b0'}, 'calibration_mxfp8_accuracy': {'filename': 'calibration_mxfp8_accuracy.json', 'size': 5689, 'sha256': '77cbe174170ea41896d26176b30a0320754eef8f62977c08f03f41e161659d26'}, 'calibration_mxfp8_performance': {'filename': 'calibration_mxfp8_performance.json', 'size': 2252, 'sha256': 'e6ad66839447d8b24b7c549fb644a547863a6277ba80d392b9d74572c420099b'}, 'calibration_affine8_accuracy': {'filename': 'calibration_affine8_accuracy.json', 'size': 5657, 'sha256': 'c1f7d2ba7a447c2fb14156e3fabda97b7577035d79bf114ea8ccacec3771ac19'}, 'calibration_affine8_performance': {'filename': 'calibration_affine8_performance.json', 'size': 2247, 'sha256': 'f1c8b7d22cc691797f9e221784f2a747386ea81c5ee1286bcd2860a1670bf89f'}}}`, limit `content-addressed inputs and fastest passing experiment` - ✅ `sensitivity_matches_calibration_dataset` — actual `{'dataset_digest': '40260fc715daa552e7b7dcd75d82441065710eccc8b90892b18826cad6f8dfcb', 'baseline_metrics_match': True}`, limit `{'dataset_digest': '40260fc715daa552e7b7dcd75d82441065710eccc8b90892b18826cad6f8dfcb', 'baseline_metrics_match': True}` - ✅ `candidate_precision_map_matches_evidence` — actual `91163d306379c513935c877d53344c628d333cf4eba2f75f0f5f00889affcd75`, limit `91163d306379c513935c877d53344c628d333cf4eba2f75f0f5f00889affcd75` - ✅ `precision_map_reproducible` — actual `c5f8b260d2e1ff2fc0405c6ef3857829ca85f49b1980a8c599fb03e06788eb1a`, limit `c5f8b260d2e1ff2fc0405c6ef3857829ca85f49b1980a8c599fb03e06788eb1a` Full per-sample evidence, sensitivity results, precision rules, PRD, ADR, and technical specification are included under `release/`, `quantization/`, and `docs/`. ## Usage Requires **Apple Silicon**. ### Primary — AX Engine **Use [AX Engine](https://github.com/defai-digital/ax-engine) as the primary inference engine** for this pack (native `unlimited_ocr`: dual vision + SWA MoE + MXFP8 + R-SWA). Quantization/release measurements for these weights used: mlx-vlm `0.6.6`, mlx `0.32.0`, python `3.14.6`. #### Install ```bash # Homebrew (recommended on macOS) brew tap defai-digital/ax-engine brew install defai-digital/ax-engine/ax-engine ax-engine doctor # or Python wheel python3 -m pip install --upgrade "ax-engine[download]>=6.11.0,<7" ``` #### Download and prepare ```bash # Managed download (writes model-manifest.json when needed) ax-engine download AutomatosX/AX-Unlimited-OCR-3B-MoE-MLX-MXFP8 # Pin a release tag (recommended for production) hf download AutomatosX/AX-Unlimited-OCR-3B-MoE-MLX-MXFP8 \ --revision v0.2.0 \ --local-dir ./AX-Unlimited-OCR-3B-MoE-MLX-MXFP8 # Ensure AX native manifest is present/valid ax-engine-bench generate-manifest ./AX-Unlimited-OCR-3B-MoE-MLX-MXFP8 --validate ``` This repository ships `model-manifest.json` so downloads are AX-ready without a manual convert step when the file is present. #### Serve ```bash ax-engine serve ./AX-Unlimited-OCR-3B-MoE-MLX-MXFP8 --port 31418 # or, after managed download of the Hub id: # ax-engine serve AutomatosX/AX-Unlimited-OCR-3B-MoE-MLX-MXFP8 --download --port 31418 ``` OpenAI-compatible routes are exposed on the local server. See the [AX Engine docs](https://github.com/defai-digital/ax-engine/blob/main/docs/GETTING-STARTED.md) for install, doctor checks, and API details. ### Alternative — mlx-vlm `mlx-vlm` can load these weights as a **secondary / alternative** path. Prefer AX Engine for AutomatosX deployments. ```bash pip install 'mlx-vlm>=0.6.4' python -m mlx_vlm.generate \ --model AutomatosX/AX-Unlimited-OCR-3B-MoE-MLX-MXFP8 \ --revision v0.2.0 \ --image document.png \ --prompt 'document parsing.' \ --max-tokens 4096 ``` ### Optional — AX-OCR document toolkit AX-OCR adds deskew, PDF, and generation profiles on top of the same Hub weights. It is a toolkit, not a replacement for AX Engine as the primary runtime. ```bash pip install 'ax-ocr[mlx]' # OCRPipeline / ax-ocr CLI: model_path=AutomatosX/AX-Unlimited-OCR-3B-MoE-MLX-MXFP8, revision=v0.2.0 ``` This is quantization/conversion work, not foundation-model retraining or domain fine-tuning.