# GR00T-N1.7-3B-p150 — Blackhole p150a vs RTX 5090 (same host, same weights, same observation)

Facts only. The GPU rows were measured on the build host during the port's golden study (2026-09-14,
`docs/research/golden-multisample.md` §3 of the port project, "N1.7 GPU timing on an idle machine"); the p150a rows are the
port's own benchmark on the shipped snapshot (Stage 2: megakernel DiT denoise) plus, for reference, the same benchmark on the
Stage-1 TTNN path of the same tree, and the served numbers of this image. Nothing here was re-measured for this file; every
number names its source.

## What was run

| | |
|---|---|
| Model (GPU) | NVIDIA's **unmodified** N1.7 policy from [Isaac-GR00T](https://github.com/NVIDIA/Isaac-GR00T) (`main`, the `n1.7-release` line): Cosmos-Reason2-2B backbone (Qwen3-VL tower, 16 Qwen3 LLM layers, deepstack) + 4 VL self-attention blocks + 32-block cross/self DiT, 4 Euler flow-matching steps — i.e. the network the p150a port is PCC-gated against |
| Weights | `nvidia/GR00T-N1.7-3B` @ `2fc962b973bccdd5d8ce4f67cc63b264d6886495` (= `tt-model.yaml` `weights.revision` = `serve.env.TT_WEIGHTS_REVISION`), from the HF cache |
| Observation | DROID sample episode 1, step 40 (`demo_data/droid_sample`): `exterior_image_1_left` and `wrist_image_left` at t−15 and t (four 320×180 frames → 4 × 112 image tokens, S = 469), 17-dim state (`eef_9d` 9 / `gripper_position` 1 / `joint_position` 7), instruction "Put the blue block in the green bowl" — the step shipped as `media/demo_observation.json` + `media/*.png` and used by the smoke test; batch 1 |
| GPU | NVIDIA GeForce RTX 5090 (sm_120), the same host as the p150a; GPU otherwise idle, 1-min load average 2.3–7.4 and falling, P1 state / 2970–2985 MHz SM (`system_state` recorded in every run JSON) |
| GPU software | Python 3.12.13, torch 2.9.0+cu128, transformers 4.57.3, flash-attn 2.8.3 — the `uv.lock` of the Isaac-GR00T checkout (`docs/research/ref-env-n17.md` §1) |
| GPU loop | one process per configuration, 20 warm-up + 20 timed iterations, `torch.cuda.synchronize()` before and after every call, medians (`golden/multi_scripts/time_n17_idle.py`) |
| GPU timing definitions | `get_action` = model only (backbone + action head); `e2e` = `get_action` + host preprocessing (albumentations chain, Qwen2-VL processor, tokenisation) + numpy decode; `backbone` and `action head` are the GPU sub-steps (`action head` = 4 DiT steps + encoders/decoder) |
| p150a (device, Stage 2 = this image's path) | one Blackhole p150a, tt-metal `668c2907575`, port snapshot `009de892c895`, the port's `benchmarks/bench_e2e.py` with its defaults (`dit_backend=megakernel`, bfp8_b arena, `mixed_dit`, `per_stage`, tuned Qwen3-VL tower, device-side text-embedding gather, 2 command queues, `worker_l1_size` 1,395,712): 10 warm-up calls, then **50 calls** of `encode → upload → 4 × execute_trace → one sync → D2H → decode`, host wall-clock medians; a second 50-call pass with a sync after every trace gives the per-trace split — `code/models/experimental/gr00t/benchmarks/results/bench_e2e_n17_20260918-050929.json` (idle host; the first run `…-050724.json`, taken while a CPU test suite was finishing, differs only in the host rows: e2e 65.8, `host_encode` 8.5) |
| p150a (device, Stage 1 reference) | the same benchmark with `--dit-backend ttnn` on the same tree the same day (`bench_e2e_n17_20260918-032335.json`: the Stage-1 TTNN denoise, firmware-default L1, adapter intermediates in L1) — what the 2026-09-17 Stage-1 package served |
| p150a (served) | this image, `POST /predict` on the demo observation, seed path, 50 requests after 5 warm-ups (`code/scripts/bench_http.py`), server-side `timing_ms` |

## Correctness reference (what "same output" means here)

The port is gated against the **fp32 run** of NVIDIA's code (the golden: SDPA, TF32 off, RTX 5090). NVIDIA's own bf16 +
flash-attention GPU path sits at a measurable distance from that golden, which bounds what any bf16 implementation can
be asked to reproduce (gates: `code/models/experimental/gr00t/tests/tt/gates/gates_multi.json`, the `multi2x` rule over the seven goldens; the canonical
thresholds in `common/golden_tap_map.json` `thresholds.n17` are its input):

| tensor | official bf16 GPU vs fp32 golden (PCC) | p150a Stage 2 (this image) vs fp32 golden (PCC) | p150a Stage 1 (2026-09-17 package) | gate |
|---|---:|---:|---:|---:|
| `vit_deepstack_mergers[0]` (first deepstack feature, 1792 × 2048) | 0.99995 | 0.999849 | 0.999849 | 0.9998 |
| `vl_self_attention_out` (DiT conditioning, 469 tokens × 2048) | 0.98592 | 0.97447 | 0.97447 | 0.95 |
| `action_pred_valid` (40 steps × 17 action dims, normalised) | 0.99996 | 0.99995 | 0.99995 | 0.997 |
| `unnormalized_actions` — `eef_9d` / `joint_position` | 0.99998 / 0.99997 | 0.99998 / 0.99996 | 0.99998 / 0.99996 | 0.9996 / 0.9998 |
| `unnormalized_actions.gripper_position` (near-constant, max\|d\|) | 0.77 (PCC, for scale) | max\|d\| 0.0031 | max\|d\| 0.0038 | abs ≤ 0.32 |
| `action_pred_normalized`, megakernel vs the Stage-1 TTNN denoise on the same device | — | ≥ 0.999992 (7 DROID samples) | — | 0.9999 (oracle gate) |

p150a Stage-2 values: `tests/tt/results/test_e2e_{untraced_taps_vs_golden,predict_actions_vs_golden}_n17_20260918-0452*.json` (the snapshot
validation run, commit `009de892c895`; bit-equal to the live tree's megakernel-default run of 04:11 the same day); the backbone / adapter taps
are bit-identical to the Stage-1 package (the megakernel does not touch them), the head taps move by ≤ 1e-5 in PCC. All 30 rows pass
(27 gated taps — the DiT's per-step internals are not observable inside one `generic_op`, so the Stage-1 path's 58-row comparison shrinks to
30 rows). On six additional DROID goldens the actions pass on five: `t2_s120` misses `llm_layer_first` (0.98998 vs 0.991) and
`joint_position` (PCC 0.99976 / max\|d\| 0.107 vs 0.9998 / 0.083) as it did on the Stage-1 path; `t1_s120` misses one megakernel-only
per-dim gate (`pred_velocity[k=2]` dim 9: 0.99588 vs 0.996; its actions pass) — see the card's Accuracy row, `SERVING.md` §4 and
`tests/tt/results/mk_default_summary.md` §3 of the port.

## GPU latency (RTX 5090, batch 1, medians of 20 iterations, ms)

| configuration (`golden-multisample.md` §3 tag) | e2e incl. host pre/post | `get_action` (model) | backbone | action head = 4 DiT steps | DiT alone, 1 step |
|---|---:|---:|---:|---:|---:|
| **official** bf16 + flash-attn 2, deterministic algorithms OFF, `cudnn.benchmark` ON, eager PyTorch (`A2_fa2_bench_plain`) | **76.45** | **63.19** | 29.43 | 33.71 | 7.91 |
| same, repeat process (`A3`) | 79.40 | 65.60 | 30.59 | 34.40 | 7.94 |
| official settings with SDPA instead of flash-attn (`B2_sdpa_bench_plain`) | 73.49 | 59.80 | 27.78 | 34.15 | 8.06 |
| official settings with `CUBLAS_WORKSPACE_CONFIG=:4096:8` set (what NVIDIA's standalone `set_seed` does; `F_fa2_bench_envonly`) | 116.65 | 104.53 | 32.19 | 72.78 | 17.07 |
| golden-dump settings: deterministic algorithms ON + the env var, `cudnn.benchmark` OFF (`D2_fa2_deterministic`) | 128.36 | 114.43 | 35.21 | 78.67 | 18.63 |

Reading: in a clean process the eager action head takes 33–35 ms and the whole model 60–66 ms; the `CUBLAS_WORKSPACE_CONFIG`
env var alone doubles the small-M DiT GEMMs (7.9 → 17.1 ms per step), which is why timings taken inside golden-dump runs are
pessimistic by ~40 ms. NVIDIA's published numbers for N1.7 (one camera): H100 eager 85.8 ms (backbone 31.3, head 48.2), **H100 /
RTX Pro 6000 TensorRT full pipeline 27.9 ms** (H100: backbone 8.8, head 12.3); third-party FlashRT on an RTX 5090 in FP8: 16.6 ms
(`docs/research/megakernel-sota.md` §8). Neither compiled nor TensorRT paths were run on this host for N1.7.

## p150a latency (batch 1, ms)

| row | Stage 2 (this image) median | p90 | Stage 1 TTNN path, same tree (reference) | source (`stages.*` of the two bench JSONs) |
|---|---:|---:|---:|---|
| `bench_e2e` end-to-end (host encode + upload + 4 traces + D2H + host decode) | **62.8** | 68.6 | 79.7 (p90 91.0) | `e2e` |
| device = upload + traces + D2H | 55.7 | 56.1 | 69.8 | `device` |
| 4 traces (one sync) | 52.4 | — | 66.1 | `traces` |
| trace `vision` (tuned Qwen3-VL tower: 24 blocks on 4 × 448 patches, 3 deepstack mergers + merger) | 17.5 | — | 17.5 | `trace/vision` (split pass) |
| trace `llm` (Qwen3 16 layers with deepstack injection, S 469 → 512) | 12.4 | — | 12.4 | `trace/llm` |
| trace `adapter` (VL-LN + 4 VL self-attention blocks, K/V hoist, state encoder; intermediates in DRAM under the megakernel) | 6.2 | — | 5.7 | `trace/adapter` |
| trace `denoise` (4 Euler steps × 32 DiT blocks + decoder) | **16.5** (4.12 per step) | — | 30.7 | `trace/denoise` |
| upload / D2H / host encode / host decode | 3.2 / 0.09 / 5.6 / 1.2 | — | 3.7 / 0.09 / 8.6 / 1.2 | `upload`, `d2h`, `host_encode`, `host_decode` |
| served `timing_ms.device` (this image, 50 requests after 5 warm-ups) | 55.9 | 56.3 | 69.5 (the Stage-1 image) | `bench_http.py` run of the container validation (`docs/publish/build-mk-n17.md`) |
| served `timing_ms.total` (handler wall incl. base64 + four PNG decodes + host encode) | 69.1 | 72.7 | 84.4 | same |
| served client wall (localhost, incl. HTTP + JSON) | 72.4 | 76.0 | 87.4 | same |

The Stage-1 column is `bench_e2e_n17_20260918-032335.json` (`--dit-backend ttnn`, same tree, same day; its `host_encode` row was taken on a
busier host) and, for the served rows, the 2026-09-17 Stage-1 package (`docs/publish/build-n17.md`). Stage 2 vs Stage 1: device −20 %,
denoise trace −46 %, e2e −21 % here and −19 % in the live tree's same-session pair (64.4 vs 79.7 ms, `tests/tt/results/mk_default_summary.md`
§5.2); the adapter trace costs +0.6 ms for its DRAM intermediates. The plan's −30 % e2e target was not reached: the denoise is now 31 % of
the traces and the backbone (vision + LLM, 29.9 ms) is the largest block.

## Comparison with matching definitions

Ratio = GPU ms / p150a ms (> 1 means the p150a is faster).

| row | p150a (Stage 2) | GPU | ratio | Stage-1 ratio (2026-09-17 package) |
|---|---:|---:|---:|---:|
| end-to-end policy call: `bench_e2e` e2e (encode + device + decode) vs eager bf16 + FA2 e2e incl. host pre/post (`A2`) | 62.8 | 76.45 | **1.22** | 1.01 |
| model only: p150a device (upload + traces + D2H) vs GPU `get_action` (`A2`) | 55.7 | 63.19 | **1.13** | 0.91 |
| denoise: p150a `denoise` trace (the megakernel) vs GPU action head (`A2`) | 16.5 | 33.71 | **2.04** | 1.10 |
| vision + LLM: p150a `vision` + `llm` traces vs GPU backbone (`A2`) | 17.5 + 12.4 = 29.9 | 29.43 | 0.98 | 0.98 |
| end-to-end vs NVIDIA's published H100 TensorRT full pipeline (one camera, different hardware and input) | 62.8 | 27.9 | GPU faster | GPU faster |

Reading: with the DiT denoise as one persistent megakernel the p150a is now ahead of NVIDIA's eager bf16 path on the same host both
end-to-end (1.22×) and model-only (1.13×), and its denoise runs in half the time of the GPU's eager action head; the backbone traces are at
parity with the GPU backbone, and the DROID layout's four 16:9 frames make the vision tower the largest p150a stage (the GR1 packages'
towers are 9–12 ms). The GPU's host pre/post (≈ 13 ms in the `A2` run) and the p150a's `host encode` + `host decode` are the same kind of
host work; HTTP/JSON framing is excluded on both sides except in the "served client wall" row.

Not measured / not claimed: p150a power (no power or efficiency comparison); `torch.compile`, TensorRT or FP8 on the GPU for
N1.7 on this host; the other N1.7 layouts (G1 4:3, three-camera). GPU numbers were not re-run for this card; they are copied
with their source tags from `golden-multisample.md` §3 (which also records the machine state of each run).