README.md

---
license: mit
base_model: baidu/Unlimited-OCR
base_model_relation: quantized
pipeline_tag: image-text-to-text
inference: false
quantized_by: shadowrock-io
library_name: vllm
metrics:
  - cer
model-index:
  - name: Unlimited-OCR-Community-FP8-Dynamic
    results:
      - task:
          type: image-to-text
          name: Grounded document OCR (parity vs BF16)
        dataset:
          name: uocr-quant synthetic document fixtures (invoice, memo, table report)
          type: shadowrock/uocr-quant-fixtures
          config: default
          split: test
        metrics:
          - type: cer_vs_bf16_mean
            name: Mean CER vs BF16 (greedy, grounding prompt)
            value: 0.0
          - type: cer_vs_bf16_max
            name: Max per-fixture CER vs BF16
            value: 0.0
          - type: decode_tok_per_s
            name: Decode throughput (tok/s, vLLM 0.26.0, RTX 5070 Ti)
            value: 39.36
        source:
          name: ShadowRock eval (raw JSON)
          url: https://huggingface.co/shadowrock-io/Unlimited-OCR-Community-FP8-Dynamic/tree/main/evidence
tags:
  - fp8
  - w8a8
  - compressed-tensors
  - llm-compressor
  - vllm
  - ocr
  - document-parsing
  - vision-language
  - moe
  - quantized
  - safetensors
  - 8-bit
language:
  - multilingual
---
<a href="https://shadowrock.io">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://a.shadowrock.team/assets/logos/full/shadowrock-logo-white.svg">
    <img alt="ShadowRock" src="https://a.shadowrock.team/assets/logos/full/shadowrock-logo-black.svg" width="340">
  </picture>
</a>

# Unlimited-OCR — Community FP8-Dynamic

**Unofficial community quantization — not a Baidu release.**

FP8 W8A8 build of [baidu/Unlimited-OCR](https://huggingface.co/baidu/Unlimited-OCR) (revision
[`07dea832`](https://huggingface.co/baidu/Unlimited-OCR/commit/07dea832e22aefee32ad281d4b80551282e1c168)), the 3.34B
DeepSeek-V2-style MoE OCR model ("one-shot long-horizon parsing"), validated against a BF16
baseline of the published checkpoint. All credit for the model belongs to Baidu; this repo changes
only the numeric precision of the text-decoder weights. Checkpoint size drops from 6.7 GB to
3.9 GB.

Pick this variant for Ada/Hopper/Blackwell GPUs where you want near-lossless behavior with native
FP8 execution. The companion
[NVFP4 build](https://huggingface.co/shadowrock-io/Unlimited-OCR-Community-NVFP4) — calibrated,
unlike prior community 4-bit builds — shrinks the footprint further for memory-constrained
Blackwell deployments.

## What is quantized

Scheme: `FP8_DYNAMIC` via [llm-compressor](https://github.com/vllm-project/llm-compressor) —
per-channel FP8 weights, dynamic per-token FP8 activations, data-free. Only the DeepSeek-V2 MoE
text-decoder linears are quantized (2196 modules: 2112 routed-expert, 33 shared-expert/dense-MLP,
48 attention, 3 dense-layer). Everything the vision path depends on stays BF16:

- SAM-ViT-B + CLIP-L DeepEncoder vision tower and projector
- `embed_tokens` / `lm_head`
- MoE router gates and all norms

## Validation

Greedy OCR on the fixture set (vLLM 0.26.0, SM120) is **character-identical to the BF16
baseline** — CER 0.0 on every fixture, grounding-box coordinates included. The same parity holds
under `transformers` + `compressed-tensors`.

| | BF16 | FP8-Dynamic (this repo) |
|---|---|---|
| Mean CER vs BF16 | — | **0.0000** |
| invoice / memo / table CER | — | 0.0 / 0.0 / 0.0 |
| Decode throughput (tok/s, greedy) | 45.3 | 39.4 |
| Checkpoint size | 6.7 GB | 3.9 GB |

Raw per-fixture transcripts ship under [`evidence/`](evidence/), produced by the
[source repo](https://git.srk.rest/shadowrock/uocr-quant) harness.

Note: the shipped `quantization_config.ignore` uses prefix-agnostic regex patterns — required for
vLLM to correctly skip the vision tower (source-repo issue #5). Do not replace it with expanded
module names.

## Usage

```python
import torch
from transformers import AutoModel, AutoTokenizer

repo = "shadowrock-io/Unlimited-OCR-Community-FP8-Dynamic"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModel.from_pretrained(repo, trust_remote_code=True,
                                  torch_dtype=torch.bfloat16, device_map="cuda").eval()
text = model.infer(tok, prompt="<image>\n<|grounding|>OCR this image.",
                   image_file="document.png", output_path="./out",
                   base_size=1024, image_size=1024, crop_mode=False, eval_mode=True)
```

## About

Quantized by [Matt Busi](https://shadowrock.io) at ShadowRock. Reproduction scripts (quantizer,
sanitizer, parity harness, fixtures) live in the
[source repo](https://git.srk.rest/shadowrock/uocr-quant). Raw evaluation outputs ship under
[`evidence/`](evidence/).