majentik/Nemotron-3-Nano-30B-A3B-TurboQuant-MLX-4bit

KV-cache quantization without any fork (recommended, 2026): upstream llama.cpp/Ollama now cover this natively — use -ctk q8_0 -ctv q8_0 (half KV memory, negligible quality loss: perplexity +0.002–0.05) or -ctk q4_0 -ctv q4_0 (quarter memory, ≈7.6% perplexity increase). In Ollama: OLLAMA_KV_CACHE_TYPE=q8_0 with OLLAMA_FLASH_ATTENTION=1. Keep K and V types symmetric to stay on the fast fused Flash-Attention path. Since April 2026, mainline llama.cpp also applies Hadamard rotation to KV activations (PR #21038), which greatly improves low-bit KV quality (opt-out: LLAMA_ATTN_ROT_DISABLE=1).

The RotorQuant/TurboQuant fork flow below is experimental/legacy: the TurboQuant llama.cpp PR was closed without merging (June 2026) and the fork is unmaintained relative to mainline. It is NOT required to use this model.

Nemotron-3-Nano-30B-A3B - TurboQuant MLX 4-bit

4-bit weight-quantized MLX version of nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 with TurboQuant KV-cache quantization. Optimized for Apple Silicon inference via the MLX framework. A good balance between model quality and memory efficiency. Only 3.2B parameters are active per token despite 30.7B total, making this model significantly more efficient at inference time than its parameter count suggests. The hybrid Mamba-2 + Transformer MoE architecture supports up to 1M context length.

Approximate model size: ~17 GB

Model Specifications

Property	Value
Base Model	nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16
Parameters	30.7 billion total (3.2 billion active per token)
Architecture	Hybrid Mamba-2 + Transformer MoE (3.2B active per token)
Context Length	1,048,576 tokens (1M)
License	NVIDIA Open Model License (commercial use OK)
Weight Quantization	4-bit (~17 GB)
KV-Cache Quantization	TurboQuant
Framework	MLX (Apple Silicon)

Quickstart

from mlx_lm import load, generate
from turboquant import TurboQuantCache

model, tokenizer = load("majentik/Nemotron-3-Nano-30B-A3B-TurboQuant-MLX-4bit")

prompt = "Explain the theory of relativity."
response = generate(model, tokenizer, prompt=prompt, max_tokens=512)
print(response)

What is TurboQuant?

TurboQuant (arXiv: 2504.19874) is a KV-cache quantization technique that compresses the key-value cache used during autoregressive generation. Combined with 4-bit weight quantization in MLX, this provides a dual compression strategy: smaller model weights plus compressed KV cache for efficient long-context generation.

Key benefits:

• No weight modification -- model weights stay at original precision
• Reduced inference memory -- KV cache is compressed significantly
• Longer context windows -- fit more tokens in the same GPU memory
• Minimal quality loss -- carefully designed quantization preserves generation quality

KV-Cache Quantization Comparison

Method	Prefill Speed	Decode Speed	Memory Savings	Reference
TurboQuant	1x (baseline)	1x (baseline)	High	arXiv: 2504.19874
RotorQuant	5.3x faster	28% faster	High	GitHub

Memory Estimates (Nemotron-3-Nano-30B-A3B)

Precision	Approximate Size	MLX Variant
BF16 (original)	~60 GB	--
8-bit quantized	~30 GB	TurboQuant-MLX-8bit
4-bit quantized	~17 GB	This model
2-bit quantized	~9 GB	TurboQuant-MLX-2bit

Hardware Requirements

This model requires approximately 17 GB of unified memory. Recommended hardware:

• Apple M2 Pro (24 GB+)
• Apple M3 Pro (24 GB+)
• Apple M4 Pro (24 GB+)
• Any Apple Silicon Mac with 24 GB+ unified memory

See Also

• nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 -- Base model
• majentik/Nemotron-3-Nano-30B-A3B-TurboQuant -- TurboQuant KV-cache only (transformers)
• majentik/Nemotron-3-Nano-30B-A3B-TurboQuant-MLX-8bit -- MLX 8-bit variant
• majentik/Nemotron-3-Nano-30B-A3B-TurboQuant-MLX-2bit -- MLX 2-bit variant
• majentik/Nemotron-3-Nano-30B-A3B-RotorQuant-MLX-4bit -- RotorQuant MLX 4-bit variant
• TurboQuant Paper (arXiv: 2504.19874)
• MLX Framework