Hugging Face's logo

sahilchachra
/
unlimited-ocr-4bit-mlx

Image-Text-to-Text
MLX
Safetensors
multilingual
deepseekocr
mlx-vlm
quantized
apple-silicon
deepseek-ocr
ocr
vision-language
multimodal
document-parsing
conversational
4-bit precision
Model card Files
xet
 Community
sahilchachra commited on
Commit
5df8010
·
verified ·
1 Parent(s): 30f3619

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +95 -43
README.md CHANGED
@@ -1,44 +1,61 @@
1
  ---
2
  license: mit
3
  base_model: baidu/Unlimited-OCR
 
 
4
  tags:
5
  - mlx
 
6
  - quantized
7
  - apple-silicon
 
 
 
 
 
 
 
8
  ---
9
 
10
- # unlimited-ocr-4bit-mlx
11
 
12
- MLX quantization of [baidu/Unlimited-OCR](https://huggingface.co/baidu/Unlimited-OCR) for Apple Silicon.
 
 
 
13
 
14
- **Variant**: Affine int4
15
- **Disk size**: 2347 MB
16
  **Quantized by**: [sahilchachra](https://huggingface.co/sahilchachra)
17
 
18
- > **Note on effective bpw**: mlx-vlm's quantizers only act on the
19
- > language tower's linear weights. The vision encoder and embeddings
20
- > stay at the source dtype (bf16), so the headline variant name
21
- > reflects the LM-tower quantization while the on-disk size averages
22
- > the two halves of the model.
23
 
24
- ## Benchmark results
 
 
 
 
25
 
26
- Evaluated on Apple M4 Pro with MLX. Model loaded once; performance and quality measured in a single pass.
27
 
28
- ### Performance
29
 
30
- | | This model | FP16 baseline |
31
  |---|---:|---:|
32
- | Decode tok/s (avg, long traces) | 252.6 | 146.2 |
33
- | Peak memory (GB) | 3.7 | 7.62 |
34
- | Disk size (MB) | 2347 | 6464 |
 
35
 
36
- ### Quality
37
 
38
- | Benchmark | This model | FP16 baseline | n |
39
- |---|---:|---:|---:|
40
- | FUNSD OCR (CER ↓) | 2.2879 | 1.7588 | 50 |
41
- | FUNSD OCR (Word F1 ↑) | 0.0 | 0.6168 | 50 |
 
 
 
42
 
43
  ## Usage
44
 
@@ -58,34 +75,69 @@ response = generate(model, processor,
58
  max_tokens=4096, verbose=True)
59
  ```
60
 
61
- > **Important — model_type mapping**: The original `baidu/Unlimited-OCR` uses
62
- > `model_type: "unlimited-ocr"` which is not directly recognized by mlx-vlm.
63
- > This quantized variant ships with `model_type` set to `"deepseekocr"` and
64
- > `processor_class` set to `"DeepseekOCRProcessor"` in the config files, which
65
- > maps to mlx-vlm's built-in DeepSeek-OCR architecture support. **No manual
66
- > patching needed** — just `load()` and go.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
67
 
68
- > If you are converting the original model yourself, update `config.json`:
69
- > `"model_type": "deepseekocr"` (was `"unlimited-ocr"`), remove the `auto_map`
70
- > block, and set `processor_config.json` → `"processor_class": "DeepseekOCRProcessor"`
71
- > (was `"UnlimitedOCRHFProcessor"`).
72
 
 
 
 
 
 
 
 
73
 
74
  ## All variants in this collection
75
 
76
- | Model | Variant |
77
- |---|---|
78
- | [sahilchachra/unlimited-ocr-4bit-mlx](https://huggingface.co/sahilchachra/unlimited-ocr-4bit-mlx) | Affine int4 ← this model |
79
- | [sahilchachra/unlimited-ocr-8bit-mlx](https://huggingface.co/sahilchachra/unlimited-ocr-8bit-mlx) | Affine int8 |
80
- | [sahilchachra/unlimited-ocr-mxfp4-mlx](https://huggingface.co/sahilchachra/unlimited-ocr-mxfp4-mlx) | Block float MX FP4 |
81
- | [sahilchachra/unlimited-ocr-mxfp8-mlx](https://huggingface.co/sahilchachra/unlimited-ocr-mxfp8-mlx) | Block float MX FP8 |
82
 
83
- ## Notes
 
 
 
 
 
84
 
85
- - Requires Apple Silicon (M1 or later) with MLX
86
- - Benchmarks run on Apple M4 Pro, 24 GB unified memory
87
- - License: see [baidu/Unlimited-OCR](https://huggingface.co/baidu/Unlimited-OCR) for the original model's license
 
 
88
 
89
- ## Original model
90
 
91
- See [baidu/Unlimited-OCR](https://huggingface.co/baidu/Unlimited-OCR) for full model details and intended use.
 
 
1
  ---
2
  license: mit
3
  base_model: baidu/Unlimited-OCR
4
+ base_model_relation: quantized
5
+ pipeline_tag: image-text-to-text
6
  tags:
7
  - mlx
8
+ - mlx-vlm
9
  - quantized
10
  - apple-silicon
11
+ - deepseek-ocr
12
+ - ocr
13
+ - vision-language
14
+ - multimodal
15
+ - document-parsing
16
+ language:
17
+ - multilingual
18
  ---
19
 
20
+ # Unlimited-OCR — MLX Affine int4 (group size 64)
21
 
22
+ MLX quantization of [**baidu/Unlimited-OCR**](https://huggingface.co/baidu/Unlimited-OCR),
23
+ a 3B vision-language OCR model that pushes **DeepSeek-OCR** one step further (one-shot,
24
+ long-horizon document parsing). This variant uses **Affine int4 (group size 64)** quantization
25
+ (5.88 effective bits/weight).
26
 
 
 
27
  **Quantized by**: [sahilchachra](https://huggingface.co/sahilchachra)
28
 
29
+ > **Note on effective bpw**: mlx-vlm's quantizers only act on the language tower's linear
30
+ > weights. The vision encoder and embeddings stay at bf16, so the on-disk size averages
31
+ > the quantized text decoder with the full-precision vision components.
 
 
32
 
33
+ ## About the model
34
+
35
+ - **Architecture:** *DeepEncoder* vision (SAM-ViT-B + CLIP-L/14, 1024×1024 input, 16× downsample) → linear projector → **DeepSeek-V2 MoE** text decoder (12 layers, hidden 1280, 64 routed + 2 shared experts, 6 experts/token).
36
+ - **Task:** multilingual OCR / document parsing — single image, multi-page, and PDF (one-shot long-horizon parsing). Supports *gundam* (crop) and *base* resolution modes.
37
+ - **License:** MIT (inherited from the base model).
38
 
39
+ ## Benchmark results
40
 
41
+ Evaluated on Apple M4 Pro (24 GB) with MLX on the [FUNSD](https://huggingface.co/datasets/nielsr/funsd) test set (50 scanned form images).
42
 
43
+ | | Affine int4 (group size 64) | FP16 baseline |
44
  |---|---:|---:|
45
+ | **FUNSD CER ↓** | **2.2879** | 1.7588 |
46
+ | Decode tok/s | 252.6 | 146.2 |
47
+ | Peak memory | 3.7 GB | 7.62 GB |
48
+ | Disk size | 2347 MB | 6464 MB |
49
 
50
+ ### All variants compared
51
 
52
+ | Variant | CER ↓ | Tok/s | Memory | Disk |
53
+ |---|---:|---:|---:|---:|
54
+ | FP16 (baseline) | 1.7588 | 146.2 | 7.62 GB | 6464 MB |
55
+ | MXFP8 | 1.4556 | 205.6 | 4.98 GB | 3660 MB |
56
+ | Int8 | 1.5720 | 205.2 | 5.06 GB | 3747 MB |
57
+ | MXFP4 | 2.3944 | 251.9 | 3.61 GB | 2260 MB |
58
+ | Int4 | 2.2879 | 252.6 | 3.7 GB | 2347 MB |
59
 
60
  ## Usage
61
 
 
75
  max_tokens=4096, verbose=True)
76
  ```
77
 
78
+ ## Prompting guide
79
+
80
+ Unlimited-OCR uses the **DeepSeek-OCR** prompt vocabulary. The prompt is just an instruction;
81
+ prefix it with `<|grounding|>` whenever you also want **bounding boxes** for what was read.
82
+
83
+ | Task | Prompt |
84
+ |---|---|
85
+ | **Document → Markdown** (layout-aware, with boxes) | `<image><|grounding|>Convert the document to markdown.` |
86
+ | **Plain text OCR** (just the text, no layout) | `<image>Free OCR.` |
87
+ | **OCR with bounding boxes** | `<image><|grounding|>OCR this image.` |
88
+ | **Native parse** | `<image>document parsing.` |
89
+ | **Parse a figure / chart / diagram** | `<image>Parse the figure.` |
90
+ | **Describe the image** (general VQA) | `<image>Describe this image in detail.` |
91
+
92
+ > **Note:** Unlike the GGUF/llama.cpp workflow, mlx-vlm requires the literal `<image>` token
93
+ > in the prompt and a separate `image=` argument pointing to the file path.
94
+
95
+ ### Understanding the output (grounding tokens)
96
+
97
+ With `<|grounding|>`, the model interleaves the recognized text with detection boxes:
98
+
99
+ ```
100
+ <|det|>title [37, 64, 464, 132]<|/det|>INVOICE #2026-0623
101
+ <|det|>text [37, 194, 350, 247]<|/det|>Bill To: Sahil Chachra
102
+ <|det|>text [37, 483, 329, 543]<|/det|>Total Due: $44.00
103
+ ```
104
+
105
+ Each `[x1, y1, x2, y2]` is the bounding box (top-left → bottom-right) of that span. Drop the
106
+ `<|det|>...<|/det|>` tags if you only want the text, or parse them to overlay boxes / build a layout.
107
+
108
+ > **Tip — long documents:** For multi-page scans, run page-by-page and concatenate.
109
+
110
+ ## Important — model_type mapping
111
 
112
+ The original `baidu/Unlimited-OCR` uses `model_type: "unlimited-ocr"` which is not directly
113
+ recognized by mlx-vlm. This quantized variant ships with the config already patched:
 
 
114
 
115
+ - `config.json` → `"model_type": "deepseekocr"` (was `"unlimited-ocr"`), `auto_map` removed
116
+ - `processor_config.json` → `"processor_class": "DeepseekOCRProcessor"` (was `"UnlimitedOCRHFProcessor"`)
117
+
118
+ **No manual patching needed** — just `load()` and go.
119
+
120
+ If you are converting the original model yourself, apply these two changes before running
121
+ `mlx_vlm convert`.
122
 
123
  ## All variants in this collection
124
 
125
+ ### MLX (Apple Silicon — this collection)
 
 
 
 
 
126
 
127
+ | Model | Variant | Disk |
128
+ |---|---|---:|
129
+ | [sahilchachra/unlimited-ocr-4bit-mlx](https://huggingface.co/sahilchachra/unlimited-ocr-4bit-mlx) | Affine int4 ← this model | 2347 MB |
130
+ | [sahilchachra/unlimited-ocr-8bit-mlx](https://huggingface.co/sahilchachra/unlimited-ocr-8bit-mlx) | Affine int8 | 3747 MB |
131
+ | [sahilchachra/unlimited-ocr-mxfp4-mlx](https://huggingface.co/sahilchachra/unlimited-ocr-mxfp4-mlx) | Block float MX FP4 | 2260 MB |
132
+ | [sahilchachra/unlimited-ocr-mxfp8-mlx](https://huggingface.co/sahilchachra/unlimited-ocr-mxfp8-mlx) | Block float MX FP8 | 3660 MB |
133
 
134
+ ### GGUF (llama.cpp — cross-platform)
135
+
136
+ | Model | Notes |
137
+ |---|---|
138
+ | [sahilchachra/Unlimited-OCR-GGUF](https://huggingface.co/sahilchachra/Unlimited-OCR-GGUF) | K-quants & i-quants (BF16 → IQ2_M). Requires llama.cpp PR #17400. |
139
 
140
+ ## Credits
141
 
142
+ - Base model: [baidu/Unlimited-OCR](https://huggingface.co/baidu/Unlimited-OCR) (MIT) — builds on [deepseek-ai/DeepSeek-OCR](https://github.com/deepseek-ai/DeepSeek-OCR).
143
+ - Quantized by [sahilchachra](https://huggingface.co/sahilchachra).