---
license: apache-2.0
library_name: mlx
pipeline_tag: image-text-to-text
tags:
- mlx
- ocr
- document-parsing
- vision-encoder-decoder
- multimodal
base_model: nvidia/NVIDIA-Nemotron-Parse-2.0
---

# Nemotron-Parse-2.0-4bit

4-bit MLX quantization of
[`nvidia/NVIDIA-Nemotron-Parse-2.0`](https://huggingface.co/nvidia/NVIDIA-Nemotron-Parse-2.0), a 903M-parameter
encoder-decoder OCR model (C-RADIO vision encoder + mBART-style decoder).
Converted with `mlx-vlm` 0.6.12 (`mlx` 0.32.0) for inference on Apple Silicon.

This is the first MLX conversion of the Nemotron-Parse architecture. The model
is **image-to-text only**: the prompt text is ignored, the image alone drives
the transcription (this matches the source implementation).

Fidelity was measured against the source, not assumed. The numbers below and
the method that produced them are in the tables that follow.

> **Loading requires the `nemotron_parse` architecture port, new in mlx-vlm
> (PR [#1866](https://github.com/Blaizzy/mlx-vlm/pull/1866)).** Until it merges, install the port branch:
> `pip install git+https://github.com/axiom-of-choice/mlx-vlm@feat/nemotron-parse`

```bash
pip install mlx-vlm
python -m mlx_vlm generate \
    --model mlx-community/Nemotron-Parse-2.0-4bit \
    --image document.png \
    --prompt "Extract the text" \
    --max-tokens 512
```

## Quantization

| | |
|---|---|
| Bits | 4 |
| Group size | 64 |
| Mode | affine |
| Effective bits/weight | **13.290** |
| Size on disk | 1.4 GB |
| Quantized tensors | 102 (decoder, group 64) |
| Unquantized tensors | 766 (vision tower + neck, bf16) |

Effective bits per weight exceeds the nominal 4 because `mlx-vlm`
quantizes only the language side and leaves the **C-RADIO vision encoder in
bf16** by design. The vision tower is 72% of all parameters, which is why the
effective figure is much higher than on a decoder-only model. Verified, not
assumed: of 766 unquantized tensors, **zero** carry a `.scales` entry.

## Measured fidelity

Weight-level metrics compare dequantized weights against the source tensor by
tensor, streamed from disk (the source ships fp32; the 8-bit SNR band for a
bf16-comparable baseline is 42.6–42.8 dB). No prompts, no sampling,
hardware-independent.

| Variant | bpw | SNR | Relative L2 | Cosine | Decode tok/s | Peak RAM |
|---|---|---|---|---|---|---|
| 4-bit | 13.290 | 20.87 dB | 9.05% | 0.995908 | 236.2 | 14.34 GB |
| 8-bit | 14.689 | 42.76 dB | 0.73% | 0.999974 | 215.5 | 14.50 GB |
| bf16 (source) | 16 | n/a | n/a | n/a | 63.7 | 14.79 GB |

Measured on an M2 Pro / 32 GB, greedy decoding.

## OCR accuracy vs ground truth

Because OCR has *ground truth*, transcription can be scored directly rather
than inferred from token distributions. Six documents were rendered locally
with exactly known content (invoice, clinical lab report, shipping label,
receipt, motor-controller datasheet, rotated receipt).

| Variant | Field accuracy | Content accuracy | Numeric recall |
|---|---|---|---|
| 4-bit | 0.8026 | 0.8026 | 0.8333 |
| 8-bit | 0.9474 | 0.9474 | 0.8914 |
| bf16 (source) | 0.9474 | 0.9474 | 0.9353 |

- **Field accuracy**: required strings present, exact match (format-sensitive).
- **Content accuracy**: the same, ignoring markup and whitespace.
- **Numeric recall**: ground-truth numbers present, ignoring markup.

Per-document field accuracy:

| Doc | bf16 | 4-bit | 8-bit |
|---|---|---|---|
| invoice | 1.0000 | 1.0000 | 1.0000 |
| lab_report | 0.8571 | 1.0000 | 0.8571 |
| shipping_label | 1.0000 | 1.0000 | 1.0000 |
| receipt | 1.0000 | 1.0000 | 1.0000 |
| spec_table | 0.8667 | 0.0000 | 0.8667 |
| rotated_receipt | 1.0000 | 1.0000 | 1.0000 |

### The 4-bit spec_table regression

At 4 bits the datasheet document transcribes as a degenerate `& \` repetition
loop (field accuracy 0.00 vs 0.87 at 8 bits). The argmax flips at the second
decoding step and the loop is self-reinforcing; NVIDIA's own generation config
(repetition_penalty 1.1, applied in this harness) does not break it. This is a
real quantization trade at 4 bits on dense table layouts — the same document
class where the model's native LaTeX output is token-heavy. Every other
document scores 1.00 at 4 bits.

The greedy path is otherwise near-tie sensitive: on `lab_report` the 4-bit
variant transcribes all fields where bf16 and 8-bit miss two (a 0.00-vs-0.05
logit margin flips the path), so per-document ordering between 4/8/bf16 is
not meaningful; the aggregate tables are.

## What was not measured

- **No distributional metrics (perplexity, top-1 agreement, KL).** The model
  is image-to-text only — text prompts are ignored — so a text-passage harness
  cannot score it.
- **No standard OCR benchmark.** No OmniDocBench, no DocVQA. The six documents
  are synthetic and rendered locally: exact ground truth, but not photographs,
  not handwriting, not skewed scans, and not non-Latin scripts.
- **No judged quality**, no layout-structure scoring, no throughput figures for
  any chip other than an M2 Pro.
- **The source's own golden generation sequence is hardware-sensitive.** The
  model's greedy path diverges between CUDA bf16 (NVIDIA's golden file) and
  CPU/Metal bf16 at the third token; the MLX port reproduces the CPU reference
  byte-for-byte. Verified against the Hugging Face implementation on the same
  inputs.

## Sibling variants

| Variant | bpw | Field accuracy |
|---|---|---|
| [Nemotron-Parse-2.0-4bit](https://huggingface.co/mlx-community/Nemotron-Parse-2.0-4bit) | 13.290 | 0.8026 |
| [Nemotron-Parse-2.0-8bit](https://huggingface.co/mlx-community/Nemotron-Parse-2.0-8bit) | 14.689 | 0.9474 |

## Credits

All credit for the underlying model belongs to
[NVIDIA](https://huggingface.co/nvidia). This is a format conversion and
quantization; no training or fine-tuning was performed. The architecture port
(`mlx_vlm/models/nemotron_parse/`) is new in mlx-vlm.