mlx-community/NVIDIA-Nemotron-3-Nano-30B-A3B-OptiQ-4bit

Built with mlx-optiq, the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. Try the Lab · All OptIQ quants · Docs

A 4-bit mixed-precision MLX quant of mlx-community/NVIDIA-Nemotron-3-Nano-30B-A3B-MLX-BF16 produced by mlx-optiq, the sensitivity-aware quantization toolkit for Apple Silicon. +2.0 Capability Score over stock uniform 4-bit, winning or tying every one of the six benchmarks.

Nemotron 3 Nano 30B-A3B is a hybrid Mamba2 + attention model with a 128-expert sparse MoE (≈3B active parameters per token). OptiQ measures each linear's KL-divergence sensitivity against a reference forward pass and assigns 4-bit or 8-bit per-layer, including the fused switch_mlp routed-expert tensors that dominate the model's parameter mass. Sensitive layers go to 8-bit; robust ones (including most of the experts) stay at 4-bit.

Quantization details

Property	Value
Predominant precision	4-bit
Layers at 8-bit (sensitive)	127
Layers at 4-bit (robust)	36
Total quantized layers	163
Achieved BPW	5.05
Group size	64
Calibration mix	six-domain mix (40 samples)
Reference for sensitivity	uniform-4-bit (bf16 doesn't fit in 36 GB RAM)
Bundled KV-cache recipe	kv_config.json, 6 attention layers @ 4-bit (4.0 avg KV bits)

We follow the same naming convention llama.cpp uses for Q4_K_M-style mixed-precision quants: the "4-bit" label is for the predominant precision, not the weighted average. Most of the 8-bit layers are the small mamba / attention projections; the big routed-expert tensors mostly stay at 4-bit, which is how the model lands at 5.05 BPW.

Usage

Load it with mlx-lm (the custom NemotronH modeling files ship in the repo and are picked up automatically):

pip install mlx-lm

from mlx_lm import load, generate

model, tokenizer = load("mlx-community/NVIDIA-Nemotron-3-Nano-30B-A3B-OptiQ-4bit")
response = generate(
    model, tokenizer,
    prompt="Explain how a sparse mixture-of-experts router decides which experts to activate.",
    max_tokens=400,
)

For mixed-precision KV-cache serving and sensitivity-aware LoRA fine-tuning, install mlx-optiq:

pip install mlx-optiq

# Serve with the bundled KV-cache recipe
optiq serve --model mlx-community/NVIDIA-Nemotron-3-Nano-30B-A3B-OptiQ-4bit \
            --kv-config kv_config.json

Benchmarks

Six-metric Capability Score (mean of MMLU + GSM8K + IFEval + BFCL + HumanEval + HashHop). Apples-to-apples comparison against stock uniform 4-bit:

Metric	OptiQ	Uniform 4-bit	Δ
MMLU (5-shot, 1000 samples)	76.2%	74.8%	+1.3
GSM8K (1000 samples, 3-shot CoT)	81.6%	78.5%	+3.1
IFEval (full set, strict)	69.1%	67.5%	+1.7
BFCL-V3 simple (200 calls)	74.0%	74.0%	+0.0
HumanEval (164 problems, pass@1)	89.0%	86.0%	+3.0
HashHop (long-context retrieval)	25.0%	22.0%	+3.0
Capability Score (mean of 6)	69.15	67.13	+2.02
On-disk size	20.6 GB	16.6 GB	+4.0

OptiQ wins or ties every benchmark. The mixed-precision allocation costs ~4 GB more on disk than stock uniform 4-bit, that disk buys a clean sweep across math, code, instruction-following, and long-context retrieval. Every metric gets one equal vote; disk size is reported next to the score as an honest second axis instead of being folded in. See the eval-framework writeup for the full methodology.

Links

• Project website: mlx-optiq.com
• PyPI: pypi.org/project/mlx-optiq
• Calibration mix: mlx-optiq.com/blog/calibration-mix
• Eval framework: mlx-optiq.com/blog/eval-framework
• Base model: mlx-community/NVIDIA-Nemotron-3-Nano-30B-A3B-MLX-BF16

Base model

This is a quantized derivative of NVIDIA Nemotron 3 Nano 30B-A3B. See the NVIDIA Open Model License for terms, the quant is distributed under the same license as the base.

Quantize your own

This quant was produced by mlx-optiq. Point it at any Hugging Face model to get the same sensitivity-aware mixed precision:

pip install mlx-optiq
optiq convert <hf-model-id> --target-bpw 5.0 --candidate-bits 4,8
optiq lab   # full local workbench: chat, compare, quantize, fine-tune