GR00T-N1.6-3B-p150 — Blackhole p150a vs RTX 5090 (same host, same weights, same observation)
Facts only. The GPU rows were measured on the build host during the port's reference-environment study
(2026-09-14, docs/research/ref-env-n16.md §5 of the port project); the p150a rows are the port's own benchmark of the
frozen snapshot this image ships (Stage-2: megakernel denoise), the same-tree TTNN-denoise run it replaced, and the served
numbers of this image. The GPU numbers were not re-measured for this file; every number names its source.
What was run
| Model (GPU) | NVIDIA's unmodified Gr00tPolicy from Isaac-GR00T n1.6.1-release — Eagle-Block2A-2B-v2 backbone (SigLIP2 252² → 81 tokens, Qwen3-1.7B[:16]) + AlternateVLDiT action head, 4 Euler flow-matching steps — i.e. the network the p150a port is PCC-gated against |
| Weights | nvidia/GR00T-N1.6-3B @ d0814e7ecb19202e7c8468b46098b0b7ef3a6d61 (= tt-model.yaml weights.revision = serve.env.TT_WEIGHTS_REVISION), from the HF cache |
| Observation | demo_data/gr1.PickNPlace episode 0, frame 0 (GR1: one 256×256 ego_view_bg_crop_pad_res256_freq20 frame, 5 state groups 7/7/6/6/3, instruction "pick the pear from the counter and place it in the plate") — the same step shipped as media/demo_observation.json + media/demo_ego_view.png and used by the smoke test; batch 1 |
| GPU | NVIDIA GeForce RTX 5090 (sm_120), the same host as the p150a; GPU otherwise idle (nvidia-smi compute-app list recorded in every run manifest) |
| GPU software | Python 3.10.20, torch 2.7.1+cu128 (cuDNN 90701), transformers 4.51.3, flash_attn 2.7.4.post1 (sm_120 SASS, imports and runs natively), numpy 1.26.4 — the uv.lock of Isaac-GR00T-n1.6 (ref-env-n16.md §1) |
| GPU loop | medians of 20 warm iterations after warm-up, torch.cuda.synchronize() around each call, one process per configuration |
| GPU timing definitions | e2e = policy.get_action incl. ~2.5 ms of CPU preprocessing (albumentations, PIL resize, tokenisation) and the numpy decode; backbone and action head are the GPU sub-steps (action head = 4 DiT steps + encoders/decoder) |
| p150a (device, Stage 2 = this image's path) | one Blackhole p150a, the frozen snapshot 009de892c895 of the port on tt-metal 668c2907575, benchmarks/bench_e2e.py defaults: dit_backend=megakernel (the DiT denoise as one persistent generic_op, bfp8_b DRAM weight arena), mixed_dit elsewhere (bf16), per_stage traces, 2 command queues with CQ-1 input uploads, device embedding gather, worker_l1_size 1,395,712: 10 warm-up calls, then 50 calls of encode → upload → 4 × execute_trace → D2H → decode, host wall clock medians; a second 50-call pass with a sync after every trace gives the per-trace split — code/models/experimental/gr00t/benchmarks/results/bench_e2e_n16_20260918-050632.json |
| p150a (device, TTNN denoise on the same tree) | the same tree and settings with --dit-backend ttnn (the Stage-1 op sequence in the denoise trace, firmware-default L1) — benchmarks/results/bench_e2e_n16_20260918-032258.json |
| p150a (served) | this image, POST /predict on the demo observation, seed path, 50 requests after 5 warm-ups (code/scripts/bench_http.py), server-side timing_ms |
Correctness reference (what "same output" means here)
The port is gated against the fp32 CPU run of NVIDIA's code (the golden). NVIDIA's own bf16 + flash-attention GPU path
sits at a measurable distance from that golden, which bounds what any bf16 implementation can be asked to reproduce
(code/models/experimental/gr00t/common/golden_tap_map.json, thresholds.n16):
| tensor | official bf16 GPU vs fp32 golden (PCC) | p150a Stage-2 (megakernel) vs fp32 golden (PCC) | p150a Stage-1 image (2026-09-15) | gate |
|---|---|---|---|---|
backbone_features (LLM output, 116 tokens × 2048) |
0.99679 | 0.99878 | 0.99878 | 0.9917 |
action_pred_valid (16 steps × 29 action dims, normalised) |
0.99956 | 0.99922 | 0.99921 | 0.998 |
unnormalized_actions — left_arm / right_arm / right_hand |
— | 0.99995 / 0.99990 / 0.99308 | 0.99995 / 0.99991 / 0.99312 | 0.9999 / 0.9998 / 0.99 |
p150a Stage-2 values: tests/tt/results/test_e2e_{untraced_taps_vs_golden,predict_actions_vs_golden}_n16_20260918-0450*.json
(the snapshot validation). Two intermediate taps of the megakernel path miss their per-tap gates on the unused state-token row
(action_decoder_out[k=1] 0.999594 vs 0.9997, action_decoder_out[k=2] 0.999598 vs 0.99978; 26/28 gated rows pass); the DiT block
intermediates (dit_out[k], two of which missed on the Stage-1 path) are not observable inside the megakernel; every action tap
passes — see the card's Caveats. Megakernel vs TTNN denoise on device (same tree, mk_default_summary.md §3.1): action_pred_normalized
PCC ≥ 0.999994 on every N1.6 reference sample.
GPU latency (RTX 5090, batch 1, medians of 20 iterations, ms)
| configuration | e2e get_action |
data proc (CPU) | backbone (GPU) | action head = 4 DiT steps (GPU) |
|---|---|---|---|---|
official: bf16 + flash-attn 2, deterministic algorithms OFF, cudnn.benchmark ON, eager PyTorch |
63.1 | 2.5 | 16.8 | 36.2 |
official + torch.compile(DiT.forward, mode="max-autotune") (as standalone_inference_script.py; compile 41 s; inductor skipped some Triton mm configs on sm_120 "out of shared memory") |
33.1 | 2.4 | 17.1 | 13.5 |
golden run bf16_fa2 (deterministic algorithms ON, CUBLAS_WORKSPACE_CONFIG=:4096:8, cudnn.benchmark OFF) |
97.3 | 2.6 | 25.3 | 66.8 |
golden run bf16_sdpa (same flags) |
93.2 | 2.9 | 23.6 | 66.9 |
golden run fp32_math (same flags, fp32, math SDPA) |
114.5 | 2.6 | 32.8 | 80.4 |
The official rows reproduce NVIDIA's README table for the RTX 5090 (eager 58 ms, compiled 37 ms e2e; backbone 18 ms; action head
38 → 16 ms). The "golden" rows are the settings the fp32 goldens were dumped with; the deterministic mode costs ~35 ms here;
on N1.7 the same CUBLAS_WORKSPACE_CONFIG=:4096:8 env var alone was shown to double the small-M DiT GEMMs
(docs/research/golden-multisample.md §3) — not separately attributed for N1.6 — and it is not a serving configuration. The eager action
head is dispatch-bound (128 DiT-block evaluations in 36 ms), which is why torch.compile gains 2.7× on it — the regime the p150a's
persistent megakernel addresses with one launch per denoise.
p150a latency (batch 1, ms)
| row | Stage 2 (this image): median | p90 | TTNN denoise, same tree: median | p90 | source |
|---|---|---|---|---|---|
bench_e2e end-to-end (host encode + upload + 4 traces + D2H + host decode) |
40.67 | 40.77 | 56.20 | 56.60 | stages.e2e of bench_e2e_n16_20260918-050632.json / …-032258.json |
| device = upload + traces + D2H | 39.17 | 39.22 | 54.39 | 54.77 | stages.device |
| 4 traces (one sync) | 37.76 | 37.78 | 52.65 | 52.70 | stages.traces |
trace vision (SigLIP2 tower + unshuffle projector) |
10.80 | 11.00 | 10.82 | 10.87 | stages.trace/vision (split pass) |
trace llm (Qwen3-1.7B[:16], S 116 → 128) |
9.55 | 9.75 | 9.55 | 9.61 | stages.trace/llm |
trace adapter (VL-LN, K/V hoist, state encoder; intermediates in DRAM under the megakernel) |
2.24 | 2.44 | 2.01 | 2.07 | stages.trace/adapter |
trace denoise (4 Euler steps × 32 DiT blocks + decoder) |
15.35 (3.84 per step) | 15.52 | 30.43 | 30.53 | stages.trace/denoise |
| upload / D2H / host encode / host decode | 1.38 / 0.04 / 1.41 / 0.10 | 1.41 / 0.05 / 1.48 / 0.10 | 1.66 / 0.07 / 1.61 / 0.17 | 1.94 / 0.09 / 1.79 / 0.23 | stages.* |
served timing_ms.device (this image, 50 requests after 5 warm-ups) |
39.6 | 39.9 | 56.8 (Stage-1 image feb77048a2ed) |
57.3 | bench_http.py run of the container validation (bench-mk-n16-c1-20260918-054614.json; Stage-1: bench-n16-c1-20260915-014844.json) |
served timing_ms.total (handler wall incl. base64 + PNG decode) |
42.1 | 42.5 | 59.4 | 59.8 | same |
| served client wall (localhost, incl. HTTP + JSON) | 44.3 | 44.8 | 61.0 | 61.6 | same |
The Stage-1 release of 2026-09-15 measured 59.42 ms e2e (benchmarks/results/e2e_stage1_n16.json: 1 command queue, host embedding
gather); the same-tree TTNN column above already carries the 2-CQ uploads and the device embedding gather that the tree gained since, so
the megakernel's own effect is the difference between the two columns: −15.5 ms e2e (−27.6 %), all of it in the denoise trace
(−15.1 ms, −49.6 %) minus the adapter's DRAM placement (+0.2 ms).
Comparison with matching definitions
Ratio = GPU ms / p150a ms (> 1 means the p150a is faster).
| row | p150a (Stage 2) | GPU | ratio |
|---|---|---|---|
end-to-end policy call: bench_e2e e2e 40.67 (encode + device + decode) vs get_action eager bf16 e2e 63.1 (data proc + backbone + head) |
40.67 | 63.1 | 1.55 (p150a faster) |
same vs get_action with torch.compile(DiT) |
40.67 | 33.1 | 0.81 (GPU 1.23× faster) |
| model only: p150a device 39.17 (upload + traces + D2H) vs GPU backbone + head 53.0 (eager) | 39.17 | 53.0 | 1.35 (p150a faster) |
| model only vs GPU compiled backbone + head 30.6 | 39.17 | 30.6 | 0.78 (GPU 1.28× faster) |
denoise: p150a denoise trace 15.35 vs GPU action head 36.2 (eager) / 13.5 (compiled) |
15.35 | 36.2 / 13.5 | 2.36 / 0.88 (GPU 1.14× faster) |
vision + LLM: p150a vision + llm traces 20.35 vs GPU backbone 16.8 |
20.35 | 16.8 | 0.83 (GPU 1.21× faster) |
Reading: against NVIDIA's eager bf16 path the Stage-2 p150a port is 1.55× faster end-to-end (the Stage-1 release was 1.06×); against
the compiled DiT the GPU is still 1.23× faster (1.80× against Stage 1). The denoise gap closed from 30.4 vs 13.5 ms to 15.35 vs 13.5 ms;
the remaining e2e gap is now the backbone (vision + llm 20.35 ms vs 16.8 ms on the GPU) plus the p150a's upload / readback, not the
DiT. The GPU's data proc (2.5 ms) and the p150a's host encode (1.4 ms) are the same kind of host work; HTTP/JSON framing is excluded on
both sides except in the "served client wall" row.
Not measured / not claimed: p150a power (no power or efficiency comparison); TensorRT or FP8 on the GPU; the compiled GPU path with
the p150a's exact host chain; N1.6's gr1_long_text layout. GPU numbers were not re-run for this card; they are copied with their
source lines from ref-env-n16.md §5 (which also records the machine state of each run).