OsaurusAI/Nemotron-3-Nano-Omni-30B-A3B-MXFP4

Nemotron-3-Nano-Omni-30B-A3B-Reasoning · MXFP4

22.6 GB · ~113 tok/s decode on M4 Max · 30B / 3B-active hybrid Mamba-2 + Attention + MoE · native MLX, zero PyTorch in the hot path

Full multimodal (text + image + audio + video) port of NVIDIA's Nemotron-3-Nano-Omni-30B-A3B-Reasoning to Apple MLX, all four modalities running natively on Metal:

Modality	Native MLX time	vs PyTorch hybrid
Text	<1s	—
Image (1 tile, 512×512)	1.4s	21× faster
Audio (20s WAV transcribe)	2.1s	15× faster
Video (8 frames analysis)	3.6s	17× faster
LLM decode	113 tok/s	identical (same tokens)

Bundle contents (single repo, everything included)

• LLM — 52-layer hybrid Mamba-2 + Attention + MoE at MXFP4 quantization (stock 4-bit affine grouped (group_size=32))
• Vision tower — NVIDIA RADIO ViT-Huge (32 blocks, 1280 hidden, 10 cls/register tokens) at fp16 — 1.31 GB
• Vision projector (mlp1) — LayerNorm + Linear + GELU + Linear → LLM hidden — 0.32 GB
• Sound encoder — parakeet (24-layer Conformer, full Transformer-XL relative-position attention) at fp16 — 1.22 GB
• Sound projector — RMSNorm + Linear + SquaredReLU + Linear → LLM hidden — 0.03 GB
• Source .py files — modeling.py, audio_model.py, image/video/audio processors (PyTorch fallback path)
• Codec sidecar (jangtq_runtime.safetensors) — codebook + Hadamard signs (JANGTQ variants only)

Quantization recipe

All 2-D Linear layers (routed experts pre-stacked, attention, shared, Mamba in/out_proj, embed, lm_head) at 4-bit affine. Mamba 1-D (A_log, D, dt_bias, conv1d) + router gate stay fp16. Loads via stock mlx_lm.load() directly.

All bundles in this family

Variant	Size	Tok/s	Loader
MXFP4	22.6 GB	~113	mlx_lm.load
JANGTQ4	19.9 GB	~82	jang_tools.load_jangtq
JANGTQ2	12.6 GB	~85	jang_tools.load_jangtq

Install

pip install jang_tools mlx mlx_lm pillow soundfile scipy librosa imageio[ffmpeg]

(Optional, for the PyTorch hybrid fallback only: pip install transformers torch torchaudio timm open_clip_torch)

Native MLX multimodal (recommended — zero PyTorch dependency)

import mlx.core as mx
from jang_tools.nemotron_omni.model import NemotronHOmni

chat = NemotronHOmni("OsaurusAI/Nemotron-3-Nano-Omni-30B-A3B-MXFP4", dtype=mx.float32)

# Text only
print(chat.turn("Capital of France?"))                                # "Paris."

# Image input — RADIO ViT runs natively in MLX on Metal (1.4s)
print(chat.turn("What's in this image?", images=["cat.jpg"]))

# Audio input — parakeet Conformer encoder native MLX (2.1s for 20s clip)
print(chat.turn("Transcribe what was said.", audio="speech.wav"))

# Video input — frame extraction + RADIO video_embedder + EVS pruning native MLX (3.6s for 8 frames)
print(chat.turn("Describe what happens.", video="clip.mp4",
                video_target_frames=8, video_apply_evs=True))

# Mixed modality
print(chat.turn(
    "Compare the image with the spoken description.",
    images=["scene.jpg"], audio="description.wav",
))

# Multi-turn — KV + Mamba state persists across turns
print(chat.turn("And what about the previous image?"))      # references prior turn
chat.reset()                                                  # new conversation

Reasoning ON / OFF

# Reasoning ON (default for Reasoning SKU): emits <think>...</think> + answer
chat.turn("Solve: 17 + 28 = ?", enable_thinking=True)

# Reasoning OFF: faster, more direct
chat.turn("Solve: 17 + 28 = ?", enable_thinking=False)

Text-only fast path (mlx_lm or load_jangtq)

For chat-only use cases, skip the multimodal load and use the LLM directly:

from mlx_lm import load, generate
model, tokenizer = load("OsaurusAI/Nemotron-3-Nano-Omni-30B-A3B-MXFP4")
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Capital of France?"}],
    tokenize=False, add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=20))   # "Paris."

(For JANGTQ4 / JANGTQ2: replace mlx_lm.load with from jang_tools.load_jangtq import load_jangtq_model. Vision/sound weights are silently dropped on the text-only path.)

Architecture (52 hybrid layers)

hybrid_override_pattern (52 chars):
"MEMEM*EMEMEM*EMEMEM*EMEMEM*EMEMEM*EMEMEMEM*EMEMEMEME"

   23 × M = Mamba-2 SSM             (state-space, O(1) cache per token)
   23 × E = MoE (128 routed × 6)     (ReLU² activation, no gate_proj)
    6 × * = Attention                 (GQA 32q / 2kv heads, NO RoPE, head_dim=128)

Multimodal towers (fp16, native MLX):
   vision_model        →  RADIO ViT (NVIDIA C-RADIOv2-H)
   mlp1                →  LayerNorm + Linear + GELU + Linear → llm_hidden
   sound_encoder       →  ParakeetEncoder (24 Conformer layers)
   sound_projection    →  RMSNorm + Linear + SquaredReLU + Linear → llm_hidden

Cache (multi-turn):
   M layers  →  MambaCache (size=2: conv state + ssm state)  O(1)/token
   * layers  →  KVCacheSimple                                  O(L)/token
   E layers  →  stateless

Native context: 262 144 tokens (no RoPE extrapolation needed)

Special tokens

Token	ID	Purpose
<image>	18	Image / video patch placeholder (video reuses )
<so_embedding>	27	Audio frame placeholder
<img> ... </img>	—	Image/video region wrapper
<sound> ... </sound>	—	Audio region wrapper
`<	im_end	>`

Sampling guidance

Mode	temperature	top_p	When
Greedy	0.0	—	Deterministic; reasoning-correct
Recommended	0.6	0.95	DeepSeek-style sampler, balanced
Avoid	1.0	1.0	At 2-bit (JANGTQ2): flat logit + quant noise → garbage tokens

Swift / vMLX support

Native Swift port is in vmlx-swift-lm under Libraries/MLXVLM/Models/NemotronHOmni/. The full multimodal pipeline (NemotronHOmni wrapper + RADIOVision + Parakeet + Projectors + image/audio/video preprocessors) compiles cleanly. Shared video utilities in Libraries/MLXVLM/VLMVideoUtils.swift are reused by Qwen 2/2.5/3/3.5/3.6 VL and Kimi VL for cross-VLM compatibility.

import MLXVLM

let frames = try await vlmExtractFramesUniform(url: videoURL, targetFrames: 32)
let pixels = vlmStackFramesIntoChannels(frames, imageSize: 512, temporalPatchDim: 2)
// → MLXArray (n_groups, T*3=6, 512, 512) for RADIO video_embedder

License

NVIDIA Open Model License — see the base model for full terms. Quantization, conversion, native MLX port, and runtime by Jinho Jang (eric@jangq.ai).

---

🦖 Osaurus is the open-source MLX inference server for Apple Silicon. 🌀 JANG is the quantization + runtime stack powering this bundle.