Unlimited-OCR-Community-NVFP4 / README.md

Community quant of baidu/Unlimited-OCR (uocr-quant)

a56ac1d verified 12 days ago

5.55 kB

license: mit
base_model: baidu/Unlimited-OCR
base_model_relation: quantized
pipeline_tag: image-text-to-text
inference: false
quantized_by: shadowrock-io
library_name: vllm
metrics:
  - cer
model-index:
  - name: Unlimited-OCR-Community-NVFP4
    results:
      - task:
          type: image-to-text
          name: Grounded document OCR (parity vs BF16)
        dataset:
          name: uocr-quant synthetic document fixtures (invoice, memo, table report)
          type: shadowrock/uocr-quant-fixtures
          config: default
          split: test
        metrics:
          - type: cer_vs_bf16_mean
            name: Mean CER vs BF16 (greedy, grounding prompt)
            value: 0.0068
          - type: cer_vs_bf16_max
            name: Max per-fixture CER vs BF16
            value: 0.0204
          - type: decode_tok_per_s
            name: Decode throughput (tok/s, vLLM 0.26.0, RTX 5070 Ti)
            value: 40.84
        source:
          name: ShadowRock eval (raw JSON)
          url: >-
            https://huggingface.co/shadowrock-io/Unlimited-OCR-Community-NVFP4/tree/main/evidence
tags:
  - nvfp4
  - fp4
  - gptq
  - compressed-tensors
  - llm-compressor
  - vllm
  - ocr
  - document-parsing
  - vision-language
  - moe
  - quantized
  - safetensors
  - 4-bit
language:
  - multilingual

Unlimited-OCR — Community NVFP4 (calibrated)

Unofficial community quantization — not a Baidu release.

Calibrated NVFP4 build of baidu/Unlimited-OCR (revision 07dea832), the 3.34B DeepSeek-V2-style MoE OCR model. All credit for the model belongs to Baidu; this repo changes only the numeric precision of the text-decoder weights. Checkpoint size drops from 6.7 GB to ~2.8 GB.

Unlike prior community 4-bit builds of this model (data-free, weight-only), this build is GPTQ-calibrated through the MoE decoder with an OCR-domain corpus (document markdown, tables, invoices, multilingual passages), so expert weights are error-compensated against realistic activation statistics, and activations carry calibrated NVFP4 global scales for native FP4 execution on Blackwell.

Pick this variant for memory-constrained Blackwell deployments (RTX 50-series, Jetson Thor, B200). For near-lossless behavior on Ada/Hopper/Blackwell, use the companion FP8-Dynamic build.

What is quantized

Scheme: NVFP4 via llm-compressor GPTQ (offloaded hessians, 32 calibration sequences, max length 2048). Only the DeepSeek-V2 MoE text-decoder linears are quantized (2196 modules). Kept in BF16:

SAM-ViT-B + CLIP-L DeepEncoder vision tower and projector
embed_tokens / lm_head
MoE router gates and all norms

Three of 768 routed-expert instances were never activated by the calibration corpus; their weights are quantized data-free from the original BF16 checkpoint and their activation scales derived from sibling-expert statistics (see source-repo issue #6 and quantization/repair_dead_experts.py). All other experts are GPTQ-calibrated.

Validation

Greedy OCR on the fixture set (vLLM 0.26.0, SM120, Marlin NVFP4 MoE backend), CER vs the BF16 baseline after stripping grounding tags:

	BF16	Data-free community NVFP4	This repo (calibrated)
Mean CER vs BF16	—	4.27*	0.0068
invoice / memo / table CER	—	unstable*	0.0 / 0.0 / 0.02
Decode throughput (tok/s, greedy)	45.3	64.2*	40.8
Checkpoint size	6.7 GB	2.8 GB	2.8 GB

* Prior community data-free build measured on the same harness: runaway repetition to the token cap on two fixtures and immediate EOS on the third — its higher tok/s reflects degenerate generation, not usable speed.

Two of three fixtures are character-identical to BF16 including box coordinates; the third differs by a single short span. See evidence/ for raw per-fixture transcripts, CER vs BF16, decode throughput, and peak VRAM from the source repo harness. Calibration corpus and provenance ship with the repo (QUANT_PROVENANCE.json).

Usage

import torch
from transformers import AutoModel, AutoTokenizer

repo = "shadowrock-io/Unlimited-OCR-Community-NVFP4"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModel.from_pretrained(repo, trust_remote_code=True,
                                  torch_dtype=torch.bfloat16, device_map="cuda").eval()
text = model.infer(tok, prompt="<image>\n<|grounding|>OCR this image.",
                   image_file="document.png", output_path="./out",
                   base_size=1024, image_size=1024, crop_mode=False, eval_mode=True)

Note: transformers loads of NVFP4 checkpoints may require TORCH_COMPILE_DISABLE=1.

About

Quantized by Matt Busi at ShadowRock. Reproduction scripts (quantizer, calibration corpus, sanitizer, parity harness, fixtures) live in the source repo. Raw evaluation outputs ship under evidence/.