GR00T-N1.5-3B-p150

NVIDIA Isaac GR00T N1.5 (3B vision-language-action policy) running on one Tenstorrent Blackhole p150a via tt-nn: one camera frame, proprioceptive state and a task instruction in, a 16-step chunk of GR1 arm and hand joint targets out. Stage-2 release: vision tower, LLM and VL adapter as TTNN ops replayed from Metal traces (bf16); the whole DiT denoise (4 steps × 16 blocks) as one persistent megakernel on 110 Tensix cores, streaming its bfp8 weights from a DRAM arena once per step. GR00T_DIT_BACKEND=ttnn selects the Stage-1 TTNN denoise. Weights: nvidia/GR00T-N1.5-3B · Paper: arXiv:2503.14734 · Upstream code: NVIDIA/Isaac-GR00T (n1.5-release) · Port: code/models/experimental/gr00t and the server code/gr00t_p150 in this repo

Runs on p150 (mesh P150).

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

Quickstart

tt-model pull  changh95/GR00T-N1.5-3B-p150 --with-weights
tt-model serve changh95/GR00T-N1.5-3B-p150

• Weights nvidia/GR00T-N1.5-3B at 869830fc749c (5.45 GB) go to your HF cache; the image does not contain them.
• Serves on port 20000 (or the next free port); ready when the log says Application startup complete.

Run with tt-cli

tt serve changh95/GR00T-N1.5-3B-p150
python3 - <<'EOF'
import base64, json
body = json.load(open("media/demo_observation.json"))   # GR1 demo step: raw joint state + instruction
body.pop("_note", None)
body["images"] = {"ego_view": base64.b64encode(open("media/demo_ego_view.png", "rb").read()).decode()}
json.dump(body, open("req.json", "w"))
EOF
curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json
tt model stop changh95/GR00T-N1.5-3B-p150

• POST /predict: images ({"ego_view": <base64 PNG/JPEG>}), state ({group: [[floats]]}: left_arm 7, right_arm 7, left_hand 6, right_hand 6, raw joint values), instruction (≤ 102 tokens); optional seed (default 0) or noise ([16][32]), return_normalized.
• GET /health, GET /info, GET /demo (the shipped demo request and its fp32 reference actions).

Response

{"actions": {"left_arm":   [[0.0032, 0.2093, 0.0567, -1.8143, -0.1239, -0.0563, 0.0177], "... 16 steps"],
             "right_arm":  [[-0.0157, -0.1731, 0.3713, -1.7457, 0.4968, -0.0574, 0.9908], "..."],
             "left_hand":  [[0.0081, 0.0121, 0.0042, 0.0064, 0.0169, 0.0380], "..."],
             "right_hand": [[-1.4850, -1.4915, -1.4734, -1.4743, -2.9550, 2.9947], "..."]},
 "action_horizon": 16, "embodiment": "gr1", "seq_len": 296, "noise_source": "client", "seed": null,
 "timing_ms": {"decode": 1.47, "encode": 2.02, "device": 33.51, "decode_actions": 0.15, "total": 37.21}}

• The example is the demo request with the reference noise, as GET /demo returns it (noise_source: "client", seed: null): the smoke response of this image's second (warm) boot, timing_ms included.
• actions are physical, un-normalised joint targets (radians), 16 steps × [7, 7, 6, 6] dims, as Gr00tPolicy.get_action returns them; timing_ms.device covers upload, the trace replays and the readback.

Demo

Input (media/demo_ego_view.png, GR1 sim PickNPlace, 256×256)	Instruction	p150a actions vs the fp32 reference (media/demo_actions.png)
	pick the pear from the counter and place it in the plate

Accuracy and speed

Metric	Value
Action chunk vs the fp32 reference (same observation, same initial noise)	PCC left_arm 0.99994 · right_arm 0.99997 · right_hand 0.99999 · left_hand max|d| 0.030 rad (near-constant group) · normalised chunk 0.99998
Intermediate taps vs the fp32 reference (the 28 taps the megakernel path exposes, from pixels; DiT intermediates not observable inside the megakernel)	30/30 rows pass (29 gated); tightest margin vit_block_first 0.99994 vs 0.9999
Megakernel vs the Stage-1 TTNN denoise, same device	normalised chunk PCC 0.999997
Traced vs untraced, determinism	bit-identical; 5 trace replays bit-identical; 2 CQs == 1 CQ
Inference on p150a (warm, batch 1, median of 50)	33.6 ms end-to-end = host encode 1.0 + upload 1.2 + traces 31.1 (vision 8.7 · LLM 8.8 · adapter 4.9 · denoise 8.9 = 2.2 per step) + readback and decode 0.2
Same tree with the Stage-1 TTNN denoise (GR00T_DIT_BACKEND=ttnn)	40.4 ms end-to-end (denoise 16.0) → megakernel −16.8 % end-to-end, −44 % denoise
Served over HTTP by this image (warm, batch 1)	32.5 ms device · 34.6 ms server-side incl. PNG decode
Same policy on an RTX 5090 (same host, NVIDIA's unmodified Gr00tPolicy, eager PyTorch bf16 + flash-attn 2)	58.0 ms end-to-end (model 55.4) → p150a 1.73× faster

Caveats

• Stage-2 path: one persistent megakernel runs the denoise (bfp8 weights streamed from a DRAM arena once per step, 110 cores, distributed LayerNorm, per-head flash SDPA); the rest is TTNN ops in three bf16 Metal traces. Its binaries (≈ 131.7 KB) fit 4.5 KB under the 136,192 B kernel-config ring, so the device runs with 64 KiB less L1 per core and the VL adapter's intermediates in DRAM (+0.4 ms); the DiT intermediates are not observable from the served model. GR00T_DIT_BACKEND=ttnn serves the Stage-1 denoise from the same image.
• Fidelity vs the Stage-1 path: gate outcomes equal on 8 of the 9 observations (the demo and 7 of the 8 multi-sample goldens); the one miss is right_arm on t3_s300_seed2, PCC 0.99987 vs the 0.9999 gate (the TTNN path passes it; the normalised chunk passes on both). The 16.8 % end-to-end gain is below the plan's 30 %.
• One embodiment and layout: GR1 arms and hands, one ego_view camera, instruction ≤ 102 tokens; anything else is refused with 400. Batch 1.
• Noise is seeded by default (seed 0, the seed the fp32 reference was dumped with) so the output is reproducible; NVIDIA's Gr00tPolicy draws fresh noise per call. Outputs are the base checkpoint's actions for the GR1 sim data config.
• Not an OpenAI-compatible API; GET /v1/models is a stub.
• First boot is cold (weight conversion + kernel JIT incl. the megakernel, READY after 2 min 14 s); later boots take about 11 s. Validated on tt-metal 668c2907575, single p150a only. Details: SERVING.md, GPU_COMPARISON.md.

Licensing

• Weights: nvidia/GR00T-N1.5-3B, NVIDIA License (non-commercial; copy in LICENSE). Not redistributed here.
• Port and serving code (code/): Apache-2.0, © 2026 Tenstorrent, distributed under the same non-commercial limitation as a derivative of the weights. Tokenizer files are from Isaac-GR00T n1.5-release (Apache-2.0).

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:

component	built from
tt-metal	668c290757550588d0ce46b180c344a462a2aaf5
code/ digest	d87906471090789d (sha256, first 16 hex digits)
built	2026-09-17T20:23:50+00:00 by tt-model 0.1.0