GR00T-N1.6-3B-p150

NVIDIA Isaac GR00T N1.6 (3B vision-language-action policy) running on one Tenstorrent Blackhole p150a via tt-nn: one camera frame, proprioceptive state and a task instruction in, a 16-step chunk of GR1 arm, hand and waist actions out. Stage-2 release: the whole DiT denoise (32 blocks × 4 flow-matching steps) runs as one persistent megakernel streaming bfp8 weights from a DRAM arena; vision, LLM and adapter are TTNN ops (bf16) replayed from Metal traces; GR00T_DIT_BACKEND=ttnn selects the Stage-1 denoise. Weights: nvidia/GR00T-N1.6-3B · Paper: arXiv:2503.14734 · Project page: gr00t-n1_6 · Upstream code: NVIDIA/Isaac-GR00T (n1.6.1-release) · Port: code/models/experimental/gr00t and the server code/gr00t_p150 in this repo

Runs on p150 (mesh P150).

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

Quickstart

tt-model pull  changh95/GR00T-N1.6-3B-p150 --with-weights
tt-model serve changh95/GR00T-N1.6-3B-p150

• Weights nvidia/GR00T-N1.6-3B at d0814e7ecb19 (6.57 GB) go to your HF cache; the image does not contain them.
• Serves on port 20000 (or the next free port); ready when the log says Application startup complete.

Run with tt-cli

tt serve changh95/GR00T-N1.6-3B-p150
python3 - <<'EOF'
import base64, json
obs = json.load(open("media/demo_observation.json"))   # GR1 demo step: raw state + instruction
obs["images"] = {"ego_view_bg_crop_pad_res256_freq20": base64.b64encode(open("media/demo_ego_view.png", "rb").read()).decode()}
json.dump(obs, open("req.json", "w"))
EOF
curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json
tt model stop changh95/GR00T-N1.6-3B-p150

• POST /predict: images ({"ego_view_bg_crop_pad_res256_freq20": <base64 PNG/JPEG>}), state ({group: [floats]}: left_arm 7, right_arm 7, left_hand 6, right_hand 6, waist 3, raw joint values), instruction (≤ 24 tokens); optional seed (default 42) or noise ([50, 128]), return_normalized.
• GET /health, GET /info (stage names the denoise backend), GET /demo (the shipped demo request and its golden actions).

Response

{"actions": {"left_arm":   [[-0.0289, 0.1453, 0.0224, -2.1383, -0.0165, -0.0225, -0.0428], "... 16 steps"],
             "right_arm":  [[-0.0024, -0.1512, 0.0269, -2.1461, -0.0091, 0.0124, 0.0044], "..."],
             "left_hand":  [[0.0135, -0.0092, -0.0047, 0.0070, 0.0455, 0.0281], "..."],
             "right_hand": [[-0.2253, -0.2421, -0.2345, -0.2311, -0.2338, 0.3385], "..."],
             "waist":      [[0.0001, -0.0060, 0.0008], "..."]},
 "action_horizon": 16, "embodiment": "gr1", "seq_len": 116, "seed": 42,
 "timing_ms": {"decode": 0.9, "encode": 1.4, "device": 39.6, "decode_actions": 0.2, "total": 42.1}}

• actions are physical, un-normalised joint targets, 16 steps × [7, 7, 6, 6, 3] dims, as Gr00tPolicy.get_action returns them for the GR1 data config (arm and hand groups are relative outputs composed onto the request's state, waist is absolute); timing_ms.device covers upload, the trace replays and the readback.

Demo

Input (media/demo_ego_view.png, GR1 sim PickNPlace, 256×256)	Instruction	p150a actions vs the fp32 reference (media/demo_actions_vs_golden.png)
	pick the pear from the counter and place it in the plate

Accuracy and speed

Metric	Value
Action chunk vs the fp32 reference (same observation, same initial noise)	PCC left_arm 0.99995 · right_arm 0.99990 · right_hand 0.99308 · left_hand / waist max|d| 0.059 / 0.013 (near-constant groups) · all 29 action dims 0.99923 (normalised valid slice 0.99922)
Intermediate taps vs the fp32 reference (the 23 taps the megakernel path exposes, from pixels; DiT block internals are not observable inside the kernel)	26 of 28 gated rows pass (29 of 31 rows incl. the determinism row and the two xfail info rows); 2 marginal misses on the DiT state-token row that no action reads (action_decoder_out[k=1,2] 0.999594 / 0.999598 vs gates 0.9997 / 0.99978), caused by bf16 error in the SigLIP2 tower; the Stage-1 card's dit_out[k=1,2] misses are now inside the kernel
Megakernel vs the Stage-1 TTNN denoise, same device	action_pred_normalized PCC ≥ 0.999994 on every N1.6 reference sample
Traced vs untraced, determinism	bit-identical (13 of 13 rows); 5 trace replays bit-identical; 2 CQs == 1 CQ
Inference on p150a (warm, batch 1, median of 50)	40.7 ms end-to-end (p90 40.8) = host encode 1.4 + upload 1.4 + traces 37.8 (vision 10.8 · LLM 9.6 · adapter 2.2 · denoise 15.3 = 3.84 ms per step) + readback and decode 0.1
Served over HTTP by this image (warm, batch 1, 50 requests)	39.6 ms device · 42.1 ms server-side incl. PNG decode (p90 42.5)
Same policy on an RTX 5090 (same host, NVIDIA's unmodified Gr00tPolicy, eager PyTorch bf16 + flash-attn 2)	63.1 ms end-to-end → p150a 1.55× faster; with torch.compile on the DiT the GPU reaches 33.1 ms → GPU 1.23× faster than this Stage-2 path

Caveats

• Stage-2 path: the denoise trace is one persistent generic_op on all 110 Tensix cores (8 readers stream the 32-block weight set once per step from a 0.98 GB bfp8 DRAM arena to 96 compute cores; distributed LayerNorm, per-head flash SDPA, Euler update and decoder inside the kernel). Against the ttnn denoise on the same tree (e2e 56.2 ms, denoise 30.4) it saves 27.6 % / 49.6 %, short of the planned 30 % e2e; GR00T_DIT_BACKEND=ttnn selects that path.
• Fidelity margin: the served chunk matches the fp32 reference at PCC 0.99923 (physical actions; 0.99922 on the normalised valid slice, where NVIDIA's own bf16 GPU run scores 0.99956), 2.3e-4 above the 0.999 boot self-check.
• Costs of the megakernel: 64 KiB less allocatable L1 per core (the kernel-config ring is 136,192 B, the compute-core binaries take ≈ 131.7 KB of it), the VL adapter's intermediates in DRAM (+0.2 ms adapter trace), and no DiT-internal taps (10 head taps + the backbone taps remain gated).
• One embodiment and layout: GR1 (embodiment slot 20), one ego_view_bg_crop_pad_res256_freq20 camera, instruction ≤ 24 tokens; anything else is refused with 400. Batch 1.
• Noise is seeded (seed 42, as the deployed policy); pass seed or noise for another draw. Outputs are the base checkpoint's actions for the GR1 sim data config.
• Not an OpenAI-compatible API; GET /v1/models is a stub.
• First boot is cold (weight conversion + kernel JIT incl. the megakernel, 58 s); later boots take 14 s. Validated on tt-metal 668c2907575, single p150a only. Details: SERVING.md, GPU_COMPARISON.md.

Licensing

• Weights: nvidia/GR00T-N1.6-3B, NVIDIA License (non-commercial; copy in LICENSE). Not redistributed here.
• Tokenizer files (code/gr00t_p150/assets/tokenizer/n16/) are from Isaac-GR00T n1.6.1-release under the same NVIDIA License.
• Port and serving code (code/): Apache-2.0, © 2026 Tenstorrent, distributed under the same non-commercial limitation as a derivative of the weights.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:

component	built from
tt-metal	668c290757550588d0ce46b180c344a462a2aaf5
code/ digest	8e6bd13a23654f91 (sha256, first 16 hex digits)
built	2026-09-17T20:40:28+00:00 by tt-model 0.1.0