John-Williams-ATL/NVIDIA-Nemotron-3-Nano-4B-BF16-ONNX

---

language: - en license: nvidia-open-model-license library_name: transformers.js tags: - transformers.js - onnx - webgpu - mamba - hybrid - text-generation - nemotron_h - in-browser - private-ai - on-device base_model: nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16

NVIDIA Nemotron-3-Nano-4B · Transformers.js / WebGPU

ONNX export of nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 for use with Transformers.js v4 and WebGPU in-browser inference.

Run a 4B hybrid Mamba-2 + Transformer reasoning model entirely in the browser — no server, no API key, no data leaving the device.

Live demo

Nemotron-3-Nano-WebGPU Space — try it now in Chrome.

---

Benchmarks — Apple M4 Mac mini (24GB unified memory)

Tested March 2026 · Chrome 123 · WebGPU · uncontested GPU · onnx-community/NVIDIA-Nemotron-3-Nano-4B-BF16-ONNX

Metric	Result
Inference speed	~25 tok/s
Time to first token	~2–3s after model load
Model load time (cold)	~45–60s (downloads ~2.5GB)
Model load time (cached)	~8–12s
JS heap usage	~16–17MB (weights live in GPU memory)
GPU memory (estimated)	~2.5GB (q4 quantised)

Note: Running any background GPU process (Python ML training, video encoding) will contend for unified memory and reduce throughput significantly. Pause background GPU tasks for clean benchmark conditions.

---

Browser compatibility

Browser	WebGPU	Status
Chrome 113+	✅	Full GPU acceleration — recommended
Edge 113+	✅	Full GPU acceleration
Firefox 141+ (Windows)	✅	Full GPU acceleration
Firefox 145+ (macOS Tahoe 26, Apple Silicon)	✅	Full GPU acceleration
Safari 18 (macOS Sequoia and earlier)	❌	WebGPU not supported — hard error
Safari 26+ (macOS Tahoe 26)	✅	Full GPU acceleration
All browsers (WASM fallback)	⚠️	CPU inference — functional but slow

Safari users on macOS Sequoia (15) and earlier: you will see Error: Unsupported device: "webgpu". Should be one of: wasm in the console. Use Chrome until you upgrade to macOS Tahoe 26.

---

Reasoning modes

Nemotron-3-Nano-4B has two inference modes controlled by a system prompt flag.

Reasoning ON (default) — the model generates a full <think>...</think> chain-of-thought trace before the final answer. Higher quality on complex tasks. The thinking trace is streamed token-by-token but may not render in all UIs until the </think> tag is emitted — this is a UI buffering behaviour, not a generation failure. Generation is proceeding normally if you see Streamed output: entries in the browser console.

Reasoning OFF — direct answer with no thinking trace. Faster, lower token count. Recommended for structured output tasks (JSON, chart data, short-form responses).

To disable reasoning, prefix your prompt with /no_think or set the system prompt:

You are a helpful assistant. /no_think

---

Usage — Transformers.js v4

Basic (Node.js / Bun / Deno)

import { pipeline } from '@huggingface/transformers';

const generator = await pipeline(
  'text-generation',
  'onnx-community/NVIDIA-Nemotron-3-Nano-4B-BF16-ONNX',
  { device: 'webgpu' }
);

// Reasoning OFF — fast structured output
const result = await generator(
  '/no_think\nGenerate a JSON object for a bar chart with 5 months of sales data.',
  { max_new_tokens: 256 }
);

console.log(result[0].generated_text);

Browser (Web Worker — recommended)

Running on the main thread will freeze the UI during model load and inference. Always use a Web Worker:

// worker.js
import { pipeline, env } from '@huggingface/transformers';

env.allowRemoteModels = true;

let generator = null;

self.addEventListener('message', async (e) => {
  const { type, prompt } = e.data;

  if (type === 'load') {
    generator = await pipeline(
      'text-generation',
      'onnx-community/NVIDIA-Nemotron-3-Nano-4B-BF16-ONNX',
      {
        device: 'webgpu',
        dtype: 'q4',
        // Stream tokens back as they generate
        streamer: (token) => self.postMessage({ type: 'token', token })
      }
    );
    self.postMessage({ type: 'ready' });
  }

  if (type === 'generate') {
    await generator(prompt, {
      max_new_tokens: 512,
      temperature: 0.6,
      top_p: 0.95
    });
    self.postMessage({ type: 'done' });
  }
});

// main.js
const worker = new Worker('./worker.js', { type: 'module' });

worker.postMessage({ type: 'load' });

worker.addEventListener('message', (e) => {
  if (e.data.type === 'token') {
    // Append token to UI
    document.getElementById('output').textContent += e.data.token;
  }
  if (e.data.type === 'ready') {
    worker.postMessage({
      type: 'generate',
      prompt: '/no_think\nYour prompt here'
    });
  }
});

Recommended temperature settings

Per NVIDIA's documentation:

• Reasoning tasks (Reasoning ON): temperature=1.0, top_p=0.95
• Tool calling / structured output (Reasoning OFF): temperature=0.6, top_p=0.95

---

Known issues

Long reasoning traces freeze demo UIs

The default Reasoning ON mode generates a full thinking trace before the answer. In some demo implementations (including the webml-community Space as of March 2026), the UI buffers the entire trace before rendering — this appears as a freeze. The model is generating correctly. Workarounds:

• Use /no_think prefix to suppress the trace for structured output tasks
• Implement a streaming UI that renders tokens as they arrive (see Web Worker example above)
• Check browser console for Streamed output: entries to confirm generation is active

Safari WebGPU not available on macOS Sequoia and earlier

Safari only supports WebGPU from macOS Tahoe 26 (the 2026 release). On current macOS, use Chrome or Edge.

Concurrent GPU workloads reduce throughput

On Apple Silicon's unified memory architecture, GPU memory is shared. A 15GB Python ML training job running in the background will leave insufficient memory for smooth inference. Pause background GPU processes before running the model.

Model weights not in JS heap

JS performance.memory reports ~16MB heap usage regardless of model size. This is correct — the ONNX WebGPU runtime manages weights directly in GPU memory outside the JS heap. Do not use JS heap metrics to diagnose memory issues with this model.

---

Model architecture

Nemotron-3-Nano-4B is a hybrid Mamba-2 + Transformer model pruned and distilled from Nemotron-Nano-9B-v2 using the Nemotron Elastic framework.

Property	Value
Parameters	4B
Architecture	Hybrid Mamba-2 + MLP + Attention (4 attention layers)
Context length	262K tokens
Attention layers	4 (sparse, for in-context reasoning anchor)
Mamba-2 layers	Majority of layers — linear complexity, no KV cache
Reasoning	ON/OFF switchable via system prompt
Languages	English + 14 additional languages
License	NVIDIA Nemotron Open Model License

The hybrid architecture is the key to in-browser viability: Mamba-2 layers maintain a fixed-size hidden state regardless of sequence length (O(1) memory per token), while the 4 sparse attention layers recover the in-context reasoning capability that pure SSMs lose. This means 262K context fits in ~2.5GB quantised — tractable for browser WebGPU.

---

Privacy characteristics

All inference runs locally on the user's device. No tokens, prompts, or outputs are transmitted to any server. Model weights are downloaded once and cached by the browser. After initial download, the model runs fully offline.

This makes Nemotron-3-Nano-4B via Transformers.js suitable for applications processing sensitive data — financial analysis, personal documents, internal business data — where cloud API inference is not acceptable.

---

System requirements

Minimum (model runs, may be slow):

• 8GB system RAM
• Any WebGPU-capable browser
• ~3GB free storage for model cache

Recommended (smooth inference at ~25 tok/s):

• 16GB+ unified memory (Apple Silicon) or dedicated VRAM
• Chrome 113+ or Edge 113+
• No competing GPU workloads
• ~3GB free storage

Tested hardware:

• Apple M4 Mac mini 24GB — 25 tok/s ✅

---

Credits

ONNX export: onnx-community (Transformers.js team)

Original model: NVIDIA — Nemotron-3-Nano-4B technical report

Benchmarks and documentation: John Williams / fxops.ai — tested March 2026 on Apple M4 Mac mini 24GB

Transformers.js v4: Joshua Lochner (Xenova) and the HuggingFace team

---

Related

• nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 — original weights (ungated)
• nvidia/NVIDIA-Nemotron-3-Nano-4B-GGUF — GGUF for llama.cpp / Ollama (ungated)
• unsloth/NVIDIA-Nemotron-3-Nano-4B-GGUF — Unsloth optimised GGUF
• Transformers.js v4 announcement
• webml-community/Nemotron-3-Nano-WebGPU — live demo Space