Instructions to use josefprusa/Nemotron-Labs-3-Puzzle-75B-A9B-oQNVFP4-MTP with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use josefprusa/Nemotron-Labs-3-Puzzle-75B-A9B-oQNVFP4-MTP with MLX:

# Make sure mlx-lm is installed
# pip install --upgrade mlx-lm

# Generate text with mlx-lm
from mlx_lm import load, generate

model, tokenizer = load("josefprusa/Nemotron-Labs-3-Puzzle-75B-A9B-oQNVFP4-MTP")

prompt = "Write a story about Einstein"
messages = [{"role": "user", "content": prompt}]
prompt = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True
)

text = generate(model, tokenizer, prompt=prompt, verbose=True)

Notebooks
Google Colab
Kaggle
Local Apps Settings
LM Studio

How to use josefprusa/Nemotron-Labs-3-Puzzle-75B-A9B-oQNVFP4-MTP with Pi:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "josefprusa/Nemotron-Labs-3-Puzzle-75B-A9B-oQNVFP4-MTP"

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "mlx-lm": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "josefprusa/Nemotron-Labs-3-Puzzle-75B-A9B-oQNVFP4-MTP"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

MLX LM

How to use josefprusa/Nemotron-Labs-3-Puzzle-75B-A9B-oQNVFP4-MTP with MLX LM:

Generate or start a chat session

# Install MLX LM
uv tool install mlx-lm
# Interactive chat REPL
mlx_lm.chat --model "josefprusa/Nemotron-Labs-3-Puzzle-75B-A9B-oQNVFP4-MTP"

Run an OpenAI-compatible server

# Install MLX LM
uv tool install mlx-lm
# Start the server
mlx_lm.server --model "josefprusa/Nemotron-Labs-3-Puzzle-75B-A9B-oQNVFP4-MTP"
# Calling the OpenAI-compatible server with curl
curl -X POST "http://localhost:8000/v1/chat/completions" \
   -H "Content-Type: application/json" \
   --data '{
     "model": "josefprusa/Nemotron-Labs-3-Puzzle-75B-A9B-oQNVFP4-MTP",
     "messages": [
       {"role": "user", "content": "Hello"}
     ]
   }'

Hermes Agent

How to use josefprusa/Nemotron-Labs-3-Puzzle-75B-A9B-oQNVFP4-MTP with Hermes Agent:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "josefprusa/Nemotron-Labs-3-Puzzle-75B-A9B-oQNVFP4-MTP"

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default josefprusa/Nemotron-Labs-3-Puzzle-75B-A9B-oQNVFP4-MTP

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use josefprusa/Nemotron-Labs-3-Puzzle-75B-A9B-oQNVFP4-MTP with OpenClaw:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "josefprusa/Nemotron-Labs-3-Puzzle-75B-A9B-oQNVFP4-MTP"

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "josefprusa/Nemotron-Labs-3-Puzzle-75B-A9B-oQNVFP4-MTP" \
  --custom-provider-id mlx-lm \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Nemotron-Labs-3-Puzzle-75B-A9B — oQNVFP4 + MTP sidecar (MLX)

Experimental, AI-built — not production software. This artifact and its serving stack were built largely by AI agents under human direction and validated by measurement, not review. Enjoy the speed, hack on it, review it before you depend on it — and if you'd like to adopt and polish it properly, please do; the author would be delighted and simply doesn't have the time.

NVIDIA's Nemotron-3 Puzzle 75B (A9B active, hybrid Mamba/attention/MoE) quantized to oQNVFP4 (NVFP4 MoE weights with per-layer time-scale sidecars, 6-bit affine elsewhere, ~48 GB) for Apple Silicon, with the checkpoint's multi-token-prediction head included as an mtp.safetensors sidecar (5.5 GB, bf16, bit-exact extraction of NVIDIA's "training-only" MTP tensors).

Most community conversions drop the MTP tensors. Kept and served, they roughly halve decode latency: the head drafts multi-token chains that the trunk verifies in a single fused pass, and acceptance on natural text is high enough (~80–86%) that speculation pays at every context depth.

Measured throughput

Apple M3 Ultra 512 GB, omlx fork, streaming API, salted natural-document prompts (no repeated-filler inflation), temperature 0, 1000-token generations at depth:

context	decode tok/s (MTP on)	decode tok/s (MTP off)	prefill tok/s
0.6k	85.0	55.0	—
2k	80.8	54.8	977
26k	63.0	—	977
53k	57.3	—	897
108k	47.5	—	771

MTP census over full runs: 2.99 tok/cycle at short context, ~1.86 at 26k+ (the adaptive depth controller shallows as verify forwards grow), acceptance stable at 80–86% throughout. Prefill is near-flat with depth — the Mamba backbone's signature.

Requirements

This artifact is not loadable by stock mlx-lm. The NVFP4 time-scale sidecars (fc1_ts/fc2_ts) and the MTP sidecar head require the omlx fork:

git clone https://github.com/josefprusa/omlx
cd omlx && uv sync --python 3.12
OMLX_WITH_CUSTOM_KERNEL=1 uv pip install -e .   # builds the native kernels (needs full Xcode)
omlx serve

Place this repository's contents in your omlx models directory, enable mtp_enabled for the model in model_settings.json, and the server logs will show [Nemotron-H MTP] sidecar head attached (pattern=*E, depth=3) plus a per-request acceptance census.

Works without MTP too (55 tok/s class): leave mtp_enabled off and the model serves as a plain checkpoint — the sidecar is simply ignored.

Provenance

Base: nvidia/Nemotron-Labs-3-Puzzle-75B-A9B (OpenMDW v1.1)
MTP tensors: bit-exact extraction from the base checkpoint's mtp.* block (bf16, unquantized) — provenance-verifiable by hash against the source shards.
Quantization: omlx oQNVFP4 pipeline (NVFP4 MoE + ts sidecars); conversion script in the fork under experimental/.

License: OpenMDW v1.1 (NVIDIA's open model license), carried unchanged in LICENSE.

Downloads last month: 125

Safetensors

Model size

75B params

Tensor type

BF16

U32

F32

MLX

Hardware compatibility

6-bit