README.md

---
license: other
license_name: nvidia-open-model-license
license_link: https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf
base_model: nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16
tags: [nemotron, multimodal, mmproj, gguf, llama.cpp, llama-mtmd, multimodal-projector]
library_name: gguf
pipeline_tag: image-text-to-text
language: [en]
datasets: [nvidia/Nemotron-Image-Training-v3]
inference: false
---

> [!NOTE]
> **Status (2026-07-07): no weights published yet.** This repository currently contains only the model card — it marks a planned variant that has not been released. Follow the repo to be notified when files land.
<!-- status-note -->

> [!TIP]
> **KV-cache quantization without any fork (recommended, 2026):** upstream
> llama.cpp/Ollama now cover this natively — use `-ctk q8_0 -ctv q8_0`
> (~half KV memory, negligible quality loss: perplexity +0.002–0.05) or
> `-ctk q4_0 -ctv q4_0` (~quarter memory, ≈7.6% perplexity increase). In
> Ollama: `OLLAMA_KV_CACHE_TYPE=q8_0` with `OLLAMA_FLASH_ATTENTION=1`. Keep
> K and V types symmetric to stay on the fast fused Flash-Attention path.
> Since April 2026, mainline llama.cpp also applies Hadamard rotation to
> KV activations ([PR #21038](https://github.com/ggml-org/llama.cpp/pull/21038)),
> which greatly improves low-bit KV quality (opt-out:
> `LLAMA_ATTN_ROT_DISABLE=1`).
>
> The RotorQuant/TurboQuant fork flow below is **experimental/legacy**: the
> TurboQuant llama.cpp PR was closed without merging (June 2026) and the fork
> is unmaintained relative to mainline. It is NOT required to use this model.
<!-- kv-upstream-note -->

# Nemotron-3-Nano-Omni-30B-A3B-Reasoning - mmproj-F16

Multimodal projector split file for `Nemotron-3-Nano-Omni-30B-A3B-Reasoning`. Required by `llama-mtmd-cli`
when running multimodal inference against any of the GGUF weight variants
in this family.

This card is a reference; the actual `mmproj-F16.gguf` binary is published
by the upstream community. For canonical NVIDIA BF16 weights see
[nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16).

## Quickstart

```bash
# This card is a reference for the multimodal projector binary used by
# llama-mtmd-cli. Pair it with any GGUF weight variant in this family:

huggingface-cli download majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-RotorQuant-GGUF-Q4_K_M Q4_K_M.gguf --local-dir ./model
huggingface-cli download majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-mmproj-F16 mmproj-F16.gguf --local-dir ./mmproj

llama-mtmd-cli \
  -m ./model/Q4_K_M.gguf \
  --mmproj ./mmproj/mmproj-F16.gguf \
  --image example.jpg \
  -p "Describe this image"
```

## Modality matrix

| Modality | Encoder | Quantization in this variant |
|---|---|---|
| Text | LLM backbone (Mamba-2 + Transformer hybrid Sparse MoE) | per the variant suffix |
| Image | CRADIO v4-H | **BF16** (kept full-precision in every non-GGUF variant; GGUF uses mmproj-F16 split file) |
| Audio | Parakeet-TDT-0.6B-v2 | **BF16** (same rationale) |
| Video | Parakeet-TDT-0.6B-v2 + frame sampler | **BF16** (≤ 2 min, 256 frames @ 2 FPS) |

NVIDIA's official FP8 / NVFP4 recipe keeps both encoders + the cross-modal
MLP projectors in BF16 to preserve multimodal accuracy. We follow that
convention in every quantized variant we ship.

## Runtime quirks

### llama.cpp

Use `llama-mtmd-cli` for multimodal inference; pass `--mmproj mmproj-F16.gguf`
(see `majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-mmproj-F16`).

**Do NOT use CUDA 13.2** — produces gibberish. Pin CUDA 12.x or
use the Metal/CPU paths.

### Ollama

Text-only; multimodal is blocked because Ollama doesn't yet support
the mmproj split-file pattern.

### Reasoning mode

`enable_thinking` defaults to `True`. To disable extended reasoning
(e.g., for latency-sensitive cases), pass `enable_thinking=False`
to the chat template / generate call. No separate "no-think"
variant card exists — this is a runtime flag, not a model variant.

## Variants in this family

(Showing 56 sibling variants under `majentik/nemotron3-nano-omni-30b-*`. The current variant — `mmproj-F16` — is **bolded**.)

| Variant | Runtime | Approx size | Use case |
|---|---|---|---|
| **mmproj-F16** | llama-mtmd-cli | ~1-2 GB | Multimodal projector (pair with any GGUF) |

## About the RotorQuant / TurboQuant labels

RotorQuant and TurboQuant are this project's **release labels**, not distinct
quantization algorithms — for any given tier, both brand repos carry
byte-identical weights produced with the standard MLX / llama.cpp quantizers.
No brand-specific speedup is claimed or measured.