nemotron-3.5-lightning-30b-a3b-nvfp4-tt-p300x2-vllm026

UNPUBLISHED TEST VARIANT on stock vLLM 0.26.0 + the standalone vllm-tt-plugin, with NVIDIA's recommended nemotron_v3 reasoning parser and qwen3_coder tool parser. NVIDIA Nemotron 3.5 Lightning 30B-A3B (NVFP4 checkpoint): a hybrid Mamba-Transformer mixture-of-experts reasoning model with a 1,048,576-token context, ported to Tenstorrent Blackhole and served through vLLM on a QuietBox2 (2x p300c, 4 chips).

Runs on p300x2 (mesh P300x2) — 1,048,576-token context, up to 32 concurrent sequences.

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

Quickstart

tt-model pull  ndaly/Nemotron-3.5-Lightning-30B-A3B-NVFP4-tt-p300x2-vllm026 --with-weights
tt-model serve ndaly/Nemotron-3.5-Lightning-30B-A3B-NVFP4-tt-p300x2-vllm026

pull --with-weights downloads the Docker image and the nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 weights at 6dbbd757ea75a8ece6e0702872e3ae53f9987728 (into your HF cache; they are not in the image). serve starts an OpenAI-compatible server on port 20000 (or the next free port, if that one is busy); the first start compiles kernels for your device, which takes several minutes, and the server is ready when it logs Application startup complete.

What was validated

Full 1M-token context on a QuietBox2 (P300x2 mesh, 4 Blackhole chips). Release validation (2026-08-19, kernel 7.0.0-28): TPOT 15.4 ms (65 tok/s/user) single-stream, TTFT 109 ms, 22/22 spec-test conformance. This container was re-verified on the same box on 2026-09-09 (kernel 7.0.0-30): identity, deterministic generation and the separate reasoning field all pass; TTFT 110 ms; single-stream TPOT measured 42 ms (24 tok/s/user) — identical to a host-native run of the bring-up environment on that day, so the gap is the host, not the container. Reasoning is returned in a separate reasoning field (deepseek_r1 parser); with chat_template_kwargs: {"enable_thinking": false} the parser has no </think> to split on and the answer lands in reasoning with empty content.

Try it

curl -s localhost:20000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4",
  "messages": [{"role": "user", "content": "Merge two sorted lists in Python."}],
  "max_tokens": 2048, "temperature": 0}'

The first boot converts the NVFP4 weights to device layout and JIT-compiles kernels (12 min on a QuietBox2); both are cached under `/.cache/tt-model//so later boots take ~4 min. After any non-clean stop, runtt-smi -r all` before serving again.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:

component	built from
tt-metal	a local checkout — commit not published (dirty tree — the image includes uncommitted changes)
vLLM	v0.26.0
vllm-tt-plugin	a local checkout — commit not published
code/ digest	5787b8dfce9d7e38 (sha256, first 16 hex digits)
built	2026-09-11T17:47:54+00:00 by tt-model 0.1.0