README.md

---
license: mit
base_model: baidu/Unlimited-OCR
base_model_relation: quantized
pipeline_tag: image-text-to-text
inference: false
quantized_by: shadowrock-io
library_name: vllm
metrics:
  - cer
model-index:
  - name: Unlimited-OCR-Community-NVFP4
    results:
      - task:
          type: image-to-text
          name: Grounded document OCR (parity vs BF16)
        dataset:
          name: uocr-quant synthetic document fixtures (invoice, memo, table report)
          type: shadowrock/uocr-quant-fixtures
          config: default
          split: test
        metrics:
          - type: cer_vs_bf16_mean
            name: Mean CER vs BF16 (greedy, grounding prompt)
            value: 0.0068
          - type: cer_vs_bf16_max
            name: Max per-fixture CER vs BF16
            value: 0.0204
          - type: decode_tok_per_s
            name: Decode throughput (tok/s, vLLM 0.26.0, RTX 5070 Ti)
            value: 40.84
        source:
          name: ShadowRock eval (raw JSON)
          url: https://huggingface.co/shadowrock-io/Unlimited-OCR-Community-NVFP4/tree/main/evidence
tags:
  - nvfp4
  - fp4
  - gptq
  - compressed-tensors
  - llm-compressor
  - vllm
  - ocr
  - document-parsing
  - vision-language
  - moe
  - quantized
  - safetensors
  - 4-bit
language:
  - multilingual
---
<a href="https://shadowrock.io">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://a.shadowrock.team/assets/logos/full/shadowrock-logo-white.svg">
    <img alt="ShadowRock" src="https://a.shadowrock.team/assets/logos/full/shadowrock-logo-black.svg" width="340">
  </picture>
</a>

# Unlimited-OCR — Community NVFP4 (calibrated)

**Unofficial community quantization — not a Baidu release.**

Calibrated NVFP4 build of [baidu/Unlimited-OCR](https://huggingface.co/baidu/Unlimited-OCR) (revision
[`07dea832`](https://huggingface.co/baidu/Unlimited-OCR/commit/07dea832e22aefee32ad281d4b80551282e1c168)), the
3.34B DeepSeek-V2-style MoE OCR model. All credit for the model belongs to Baidu; this repo
changes only the numeric precision of the text-decoder weights. Checkpoint size drops from
6.7 GB to ~2.8 GB.

Unlike prior community 4-bit builds of this model (data-free, weight-only), this build is
**GPTQ-calibrated through the MoE decoder** with an OCR-domain corpus (document markdown, tables,
invoices, multilingual passages), so expert weights are error-compensated against realistic
activation statistics, and activations carry calibrated NVFP4 global scales for native FP4
execution on Blackwell.

Pick this variant for memory-constrained Blackwell deployments (RTX 50-series, Jetson Thor,
B200). For near-lossless behavior on Ada/Hopper/Blackwell, use the companion
[FP8-Dynamic build](https://huggingface.co/shadowrock-io/Unlimited-OCR-Community-FP8-Dynamic).

## What is quantized

Scheme: NVFP4 via [llm-compressor](https://github.com/vllm-project/llm-compressor) GPTQ
(offloaded hessians, 32 calibration sequences, max length 2048). Only the DeepSeek-V2 MoE
text-decoder linears are quantized (2196 modules). Kept in BF16:

- SAM-ViT-B + CLIP-L DeepEncoder vision tower and projector
- `embed_tokens` / `lm_head`
- MoE router gates and all norms

Three of 768 routed-expert instances were never activated by the calibration corpus; their
weights are quantized data-free from the original BF16 checkpoint and their activation scales
derived from sibling-expert statistics (see source-repo issue #6 and
`quantization/repair_dead_experts.py`). All other experts are GPTQ-calibrated.

## Validation

Greedy OCR on the fixture set (vLLM 0.26.0, SM120, Marlin NVFP4 MoE backend), CER vs the BF16
baseline after stripping grounding tags:

| | BF16 | Data-free community NVFP4 | **This repo (calibrated)** |
|---|---|---|---|
| Mean CER vs BF16 | — | 4.27* | **0.0068** |
| invoice / memo / table CER | — | unstable* | 0.0 / 0.0 / 0.02 |
| Decode throughput (tok/s, greedy) | 45.3 | 64.2* | 40.8 |
| Checkpoint size | 6.7 GB | 2.8 GB | 2.8 GB |

\* Prior community data-free build measured on the same harness: runaway repetition to the token
cap on two fixtures and immediate EOS on the third — its higher tok/s reflects degenerate
generation, not usable speed.

Two of three fixtures are character-identical to BF16 including box coordinates; the third
differs by a single short span. See `evidence/` for raw
per-fixture transcripts, CER vs BF16, decode throughput, and peak VRAM from the
[source repo](https://git.srk.rest/shadowrock/uocr-quant) harness. Calibration corpus and
provenance ship with the repo (`QUANT_PROVENANCE.json`).

## Usage

```python
import torch
from transformers import AutoModel, AutoTokenizer

repo = "shadowrock-io/Unlimited-OCR-Community-NVFP4"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModel.from_pretrained(repo, trust_remote_code=True,
                                  torch_dtype=torch.bfloat16, device_map="cuda").eval()
text = model.infer(tok, prompt="<image>\n<|grounding|>OCR this image.",
                   image_file="document.png", output_path="./out",
                   base_size=1024, image_size=1024, crop_mode=False, eval_mode=True)
```

Note: `transformers` loads of NVFP4 checkpoints may require `TORCH_COMPILE_DISABLE=1`.

## About

Quantized by [Matt Busi](https://shadowrock.io) at ShadowRock. Reproduction scripts (quantizer,
calibration corpus, sanitizer, parity harness, fixtures) live in the
[source repo](https://git.srk.rest/shadowrock/uocr-quant). Raw evaluation outputs ship under
[`evidence/`](evidence/).