README.md

metadata

license: mit
base_model: baidu/Unlimited-OCR
base_model_relation: quantized
pipeline_tag: image-text-to-text
inference: false
quantized_by: shadowrock-io
library_name: vllm
metrics:
  - cer
model-index:
  - name: Unlimited-OCR-Community-FP8-Dynamic
    results:
      - task:
          type: image-to-text
          name: Grounded document OCR (parity vs BF16)
        dataset:
          name: uocr-quant synthetic document fixtures (invoice, memo, table report)
          type: shadowrock/uocr-quant-fixtures
          config: default
          split: test
        metrics:
          - type: cer_vs_bf16_mean
            name: Mean CER vs BF16 (greedy, grounding prompt)
            value: 0
          - type: cer_vs_bf16_max
            name: Max per-fixture CER vs BF16
            value: 0
          - type: decode_tok_per_s
            name: Decode throughput (tok/s, vLLM 0.26.0, RTX 5070 Ti)
            value: 39.36
        source:
          name: ShadowRock eval (raw JSON)
          url: >-
            https://huggingface.co/shadowrock-io/Unlimited-OCR-Community-FP8-Dynamic/tree/main/evidence
tags:
  - fp8
  - w8a8
  - compressed-tensors
  - llm-compressor
  - vllm
  - ocr
  - document-parsing
  - vision-language
  - moe
  - quantized
  - safetensors
  - 8-bit
language:
  - multilingual

Unlimited-OCR — Community FP8-Dynamic

Unofficial community quantization — not a Baidu release.

FP8 W8A8 build of baidu/Unlimited-OCR (revision 07dea832), the 3.34B DeepSeek-V2-style MoE OCR model ("one-shot long-horizon parsing"), validated against a BF16 baseline of the published checkpoint. All credit for the model belongs to Baidu; this repo changes only the numeric precision of the text-decoder weights. Checkpoint size drops from 6.7 GB to 3.9 GB.

Pick this variant for Ada/Hopper/Blackwell GPUs where you want near-lossless behavior with native FP8 execution. The companion NVFP4 build — calibrated, unlike prior community 4-bit builds — shrinks the footprint further for memory-constrained Blackwell deployments.

What is quantized

Scheme: FP8_DYNAMIC via llm-compressor — per-channel FP8 weights, dynamic per-token FP8 activations, data-free. Only the DeepSeek-V2 MoE text-decoder linears are quantized (2196 modules: 2112 routed-expert, 33 shared-expert/dense-MLP, 48 attention, 3 dense-layer). Everything the vision path depends on stays BF16:

• SAM-ViT-B + CLIP-L DeepEncoder vision tower and projector
• embed_tokens / lm_head
• MoE router gates and all norms

Validation

Greedy OCR on the fixture set (vLLM 0.26.0, SM120) is character-identical to the BF16 baseline — CER 0.0 on every fixture, grounding-box coordinates included. The same parity holds under transformers + compressed-tensors.

	BF16	FP8-Dynamic (this repo)
Mean CER vs BF16	—	0.0000
invoice / memo / table CER	—	0.0 / 0.0 / 0.0
Decode throughput (tok/s, greedy)	45.3	39.4
Checkpoint size	6.7 GB	3.9 GB

Raw per-fixture transcripts ship under evidence/, produced by the source repo harness.

Note: the shipped quantization_config.ignore uses prefix-agnostic regex patterns — required for vLLM to correctly skip the vision tower (source-repo issue #5). Do not replace it with expanded module names.

Usage

import torch
from transformers import AutoModel, AutoTokenizer

repo = "shadowrock-io/Unlimited-OCR-Community-FP8-Dynamic"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModel.from_pretrained(repo, trust_remote_code=True,
                                  torch_dtype=torch.bfloat16, device_map="cuda").eval()
text = model.infer(tok, prompt="<image>\n<|grounding|>OCR this image.",
                   image_file="document.png", output_path="./out",
                   base_size=1024, image_size=1024, crop_mode=False, eval_mode=True)

About

Quantized by Matt Busi at ShadowRock. Reproduction scripts (quantizer, sanitizer, parity harness, fixtures) live in the source repo. Raw evaluation outputs ship under evidence/.