File size: 4,785 Bytes
c515c33 6112d3e c515c33 6112d3e c515c33 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 | ---
license: mit
base_model: baidu/Unlimited-OCR
base_model_relation: quantized
pipeline_tag: image-text-to-text
inference: false
quantized_by: shadowrock-io
library_name: vllm
metrics:
- cer
model-index:
- name: Unlimited-OCR-Community-FP8-Dynamic
results:
- task:
type: image-to-text
name: Grounded document OCR (parity vs BF16)
dataset:
name: uocr-quant synthetic document fixtures (invoice, memo, table report)
type: shadowrock/uocr-quant-fixtures
config: default
split: test
metrics:
- type: cer_vs_bf16_mean
name: Mean CER vs BF16 (greedy, grounding prompt)
value: 0.0
- type: cer_vs_bf16_max
name: Max per-fixture CER vs BF16
value: 0.0
- type: decode_tok_per_s
name: Decode throughput (tok/s, vLLM 0.26.0, RTX 5070 Ti)
value: 39.36
source:
name: ShadowRock eval (raw JSON)
url: https://huggingface.co/shadowrock-io/Unlimited-OCR-Community-FP8-Dynamic/tree/main/evidence
tags:
- fp8
- w8a8
- compressed-tensors
- llm-compressor
- vllm
- ocr
- document-parsing
- vision-language
- moe
- quantized
- safetensors
- 8-bit
language:
- multilingual
---
<a href="https://shadowrock.io">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://a.shadowrock.team/assets/logos/full/shadowrock-logo-white.svg">
<img alt="ShadowRock" src="https://a.shadowrock.team/assets/logos/full/shadowrock-logo-black.svg" width="340">
</picture>
</a>
# Unlimited-OCR — Community FP8-Dynamic
**Unofficial community quantization — not a Baidu release.**
FP8 W8A8 build of [baidu/Unlimited-OCR](https://huggingface.co/baidu/Unlimited-OCR) (revision
[`07dea832`](https://huggingface.co/baidu/Unlimited-OCR/commit/07dea832e22aefee32ad281d4b80551282e1c168)), the 3.34B
DeepSeek-V2-style MoE OCR model ("one-shot long-horizon parsing"), validated against a BF16
baseline of the published checkpoint. All credit for the model belongs to Baidu; this repo changes
only the numeric precision of the text-decoder weights. Checkpoint size drops from 6.7 GB to
3.9 GB.
Pick this variant for Ada/Hopper/Blackwell GPUs where you want near-lossless behavior with native
FP8 execution. The companion
[NVFP4 build](https://huggingface.co/shadowrock-io/Unlimited-OCR-Community-NVFP4) — calibrated,
unlike prior community 4-bit builds — shrinks the footprint further for memory-constrained
Blackwell deployments.
## What is quantized
Scheme: `FP8_DYNAMIC` via [llm-compressor](https://github.com/vllm-project/llm-compressor) —
per-channel FP8 weights, dynamic per-token FP8 activations, data-free. Only the DeepSeek-V2 MoE
text-decoder linears are quantized (2196 modules: 2112 routed-expert, 33 shared-expert/dense-MLP,
48 attention, 3 dense-layer). Everything the vision path depends on stays BF16:
- SAM-ViT-B + CLIP-L DeepEncoder vision tower and projector
- `embed_tokens` / `lm_head`
- MoE router gates and all norms
## Validation
Greedy OCR on the fixture set (vLLM 0.26.0, SM120) is **character-identical to the BF16
baseline** — CER 0.0 on every fixture, grounding-box coordinates included. The same parity holds
under `transformers` + `compressed-tensors`.
| | BF16 | FP8-Dynamic (this repo) |
|---|---|---|
| Mean CER vs BF16 | — | **0.0000** |
| invoice / memo / table CER | — | 0.0 / 0.0 / 0.0 |
| Decode throughput (tok/s, greedy) | 45.3 | 39.4 |
| Checkpoint size | 6.7 GB | 3.9 GB |
Raw per-fixture transcripts ship under [`evidence/`](evidence/), produced by the
[source repo](https://git.srk.rest/shadowrock/uocr-quant) harness.
Note: the shipped `quantization_config.ignore` uses prefix-agnostic regex patterns — required for
vLLM to correctly skip the vision tower (source-repo issue #5). Do not replace it with expanded
module names.
## Usage
```python
import torch
from transformers import AutoModel, AutoTokenizer
repo = "shadowrock-io/Unlimited-OCR-Community-FP8-Dynamic"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModel.from_pretrained(repo, trust_remote_code=True,
torch_dtype=torch.bfloat16, device_map="cuda").eval()
text = model.infer(tok, prompt="<image>\n<|grounding|>OCR this image.",
image_file="document.png", output_path="./out",
base_size=1024, image_size=1024, crop_mode=False, eval_mode=True)
```
## About
Quantized by [Matt Busi](https://shadowrock.io) at ShadowRock. Reproduction scripts (quantizer,
sanitizer, parity harness, fixtures) live in the
[source repo](https://git.srk.rest/shadowrock/uocr-quant). Raw evaluation outputs ship under
[`evidence/`](evidence/).
|