# GR00T-N1.6-3B-p150 — Blackhole p150a vs RTX 5090 (same host, same weights, same observation)

Facts only. The GPU rows were measured on the build host during the port's reference-environment study
(2026-09-14, `docs/research/ref-env-n16.md` §5 of the port project); the p150a rows are the port's own Stage-1
benchmark and the served numbers of this image. Nothing here was re-measured for this file; every number names its source.

## What was run

| | |
|---|---|
| Model (GPU) | NVIDIA's **unmodified** `Gr00tPolicy` from [Isaac-GR00T](https://github.com/NVIDIA/Isaac-GR00T) `n1.6.1-release` — Eagle-Block2A-2B-v2 backbone (SigLIP2 252² → 81 tokens, Qwen3-1.7B[:16]) + `AlternateVLDiT` action head, 4 Euler flow-matching steps — i.e. the network the p150a port is PCC-gated against |
| Weights | `nvidia/GR00T-N1.6-3B` @ `d0814e7ecb19202e7c8468b46098b0b7ef3a6d61` (= `tt-model.yaml` `weights.revision` = `serve.env.TT_WEIGHTS_REVISION`), from the HF cache |
| Observation | `demo_data/gr1.PickNPlace` episode 0, frame 0 (GR1: one 256×256 `ego_view_bg_crop_pad_res256_freq20` frame, 5 state groups 7/7/6/6/3, instruction "pick the pear from the counter and place it in the plate") — the same step shipped as `media/demo_observation.json` + `media/demo_ego_view.png` and used by the smoke test; batch 1 |
| GPU | NVIDIA GeForce RTX 5090 (sm_120), the same host as the p150a; GPU otherwise idle (`nvidia-smi` compute-app list recorded in every run manifest) |
| GPU software | Python 3.10.20, torch 2.7.1+cu128 (cuDNN 90701), transformers 4.51.3, flash_attn 2.7.4.post1 (sm_120 SASS, imports and runs natively), numpy 1.26.4 — the `uv.lock` of `Isaac-GR00T-n1.6` (`ref-env-n16.md` §1) |
| GPU loop | medians of **20 warm iterations** after warm-up, `torch.cuda.synchronize()` around each call, one process per configuration |
| GPU timing definitions | `e2e` = `policy.get_action` incl. ~2.5 ms of CPU preprocessing (albumentations, PIL resize, tokenisation) and the numpy decode; `backbone` and `action head` are the GPU sub-steps (`action head` = 4 DiT steps + encoders/decoder) |
| p150a (device) | one Blackhole p150a, tt-metal `668c2907575`, the port's `benchmarks/bench_e2e.py` (Stage-1: TTNN ops in four Metal traces, `mixed_dit` = bfp8_b DiT matmul weights / bf16 elsewhere, `per_stage` layout): 10 warm-up calls, then **50 calls** of `encode → upload (+sync) → 4 × execute_trace → one sync → D2H → decode`, host wall clock medians; a second 50-call pass with a sync after every trace gives the per-trace split — `code/models/experimental/gr00t/benchmarks/results/e2e_stage1_n16.json` |
| p150a (served) | this image, `POST /predict` on the demo observation, seed path, 50 requests after 5 warm-ups (`code/scripts/bench_http.py`), server-side `timing_ms` |

## Correctness reference (what "same output" means here)

The port is gated against the **fp32 CPU run** of NVIDIA's code (the golden). NVIDIA's own bf16 + flash-attention GPU path
sits at a measurable distance from that golden, which bounds what any bf16 implementation can be asked to reproduce
(`code/models/experimental/gr00t/common/golden_tap_map.json`, `thresholds.n16`):

| tensor | official bf16 GPU vs fp32 golden (PCC) | p150a Stage-1 vs fp32 golden (PCC) | gate |
|---|---:|---:|---:|
| `backbone_features` (LLM output, 116 tokens × 2048) | 0.99679 | 0.99878 | 0.9917 |
| `action_pred_valid` (16 steps × 29 action dims, normalised) | 0.99956 | 0.99921 | 0.998 |
| `unnormalized_actions` — `left_arm` / `right_arm` / `right_hand` | — | 0.99995 / 0.99991 / 0.99312 | 0.9999 / 0.9998 / 0.99 |

p150a values: `tests/tt/results/test_e2e_{untraced_taps_vs_golden,predict_actions_vs_golden}_n16_20260914-2324*.json`. Four
intermediate DiT taps of the p150a run miss their per-tap gates on the unused state-token row (`dit_out[k=1]` 0.999556 vs 0.99961,
`dit_out[k=2]` 0.999496 vs 0.99964, `action_decoder_out[k=1]` 0.999579 vs 0.9997, `action_decoder_out[k=2]` 0.999556 vs 0.99978);
every action tap passes — see the card's Caveats.

## GPU latency (RTX 5090, batch 1, medians of 20 iterations, ms)

| configuration | e2e `get_action` | data proc (CPU) | backbone (GPU) | action head = 4 DiT steps (GPU) |
|---|---:|---:|---:|---:|
| **official**: bf16 + flash-attn 2, deterministic algorithms OFF, `cudnn.benchmark` ON, eager PyTorch | **63.1** | 2.5 | 16.8 | 36.2 |
| **official + `torch.compile(DiT.forward, mode="max-autotune")`** (as `standalone_inference_script.py`; compile 41 s; inductor skipped some Triton mm configs on sm_120 "out of shared memory") | **33.1** | 2.4 | 17.1 | 13.5 |
| golden run `bf16_fa2` (deterministic algorithms ON, `CUBLAS_WORKSPACE_CONFIG=:4096:8`, `cudnn.benchmark` OFF) | 97.3 | 2.6 | 25.3 | 66.8 |
| golden run `bf16_sdpa` (same flags) | 93.2 | 2.9 | 23.6 | 66.9 |
| golden run `fp32_math` (same flags, fp32, math SDPA) | 114.5 | 2.6 | 32.8 | 80.4 |

The official rows reproduce NVIDIA's README table for the RTX 5090 (eager 58 ms, compiled 37 ms e2e; backbone 18 ms; action head
38 → 16 ms). The "golden" rows are the settings the fp32 goldens were dumped with; the deterministic mode costs ~35 ms
(`CUBLAS_WORKSPACE_CONFIG` roughly doubles the small-M DiT GEMMs — `docs/research/golden-multisample.md` §3) and is not a serving
configuration. The eager action head is dispatch-bound (128 DiT-block evaluations in 36 ms), which is why `torch.compile` gains 2.7×
on it — and the regime the port's Stage-2 persistent megakernel targets.

## p150a latency (batch 1, ms)

| row | median | p90 | source |
|---|---:|---:|---|
| `bench_e2e` end-to-end (host encode + upload + 4 traces + D2H + host decode) | **59.42** | 60.10 | `e2e_stage1_n16.json` `stages.e2e` |
| device = upload + traces + D2H | 57.20 | 57.67 | `stages.device` |
| 4 traces (one sync) | 53.27 | 53.29 | `stages.traces` |
| trace `vision` (SigLIP2 tower + unshuffle projector) | 11.49 | 11.50 | `stages.trace/vision` (split pass) |
| trace `llm` (Qwen3-1.7B[:16], S 116 → 128) | 9.50 | 9.52 | `stages.trace/llm` |
| trace `adapter` (VL-LN, K/V hoist, state encoder) | 1.98 | 1.99 | `stages.trace/adapter` |
| trace `denoise` (4 Euler steps × 32 DiT blocks + decoder) | 30.40 | 30.41 | `stages.trace/denoise` |
| upload / D2H / host encode / host decode | 3.79 / 0.09 / 1.99 / 0.17 | 4.31 / 0.11 / 2.28 / 0.21 | `stages.*` |
| served `timing_ms.device` (this image, 50 requests after 5 warm-ups, quiet host) | 56.8 | 57.3 | `bench_http.py` run of the container validation (round 1, `bench-n16-c1-20260915-014844.json`) |
| served `timing_ms.total` (handler wall incl. base64 + PNG decode) | 59.4 | 59.8 | same |
| served client wall (localhost, incl. HTTP + JSON) | 61.0 | 61.6 | same |

## Comparison with matching definitions

Ratio = GPU ms / p150a ms (> 1 means the p150a is faster).

| row | p150a | GPU | ratio |
|---|---:|---:|---:|
| end-to-end policy call: `bench_e2e` e2e 59.42 (encode + device + decode) vs `get_action` eager bf16 e2e 63.1 (data proc + backbone + head) | 59.42 | 63.1 | **1.06** (p150a faster) |
| same vs `get_action` with `torch.compile(DiT)` | 59.42 | 33.1 | **0.56** (GPU 1.80× faster) |
| model only: p150a device 57.20 (upload + traces + D2H) vs GPU backbone + head 53.0 (eager) | 57.20 | 53.0 | 0.93 (GPU 1.08× faster) |
| model only vs GPU compiled backbone + head 30.6 | 57.20 | 30.6 | 0.53 (GPU 1.87× faster) |
| denoise: p150a `denoise` trace 30.40 vs GPU action head 36.2 (eager) / 13.5 (compiled) | 30.40 | 36.2 / 13.5 | 1.19 / 0.44 |
| vision + LLM: p150a `vision` + `llm` traces 20.99 vs GPU backbone 16.8 | 20.99 | 16.8 | 0.80 (GPU 1.25× faster) |

Reading: against NVIDIA's eager bf16 path the Stage-1 p150a port is at parity end-to-end (1.06× faster); against the compiled
DiT the GPU is 1.8× faster, entirely in the denoise stage (30.4 vs 13.5 ms) — the p150a's four-step DiT is the part the
persistent megakernel (Stage 2, not in this image; weight streaming measured at 464 GB/s bf16 / 414 GB/s bfp8 in
`tests/tt/results/mk_k1_summary.md`) is meant to replace. The GPU's `data proc` (2.5 ms) and the p150a's `host encode` (2.0 ms) are the
same kind of host work; HTTP/JSON framing is excluded on both sides except in the "served client wall" row.

Not measured / not claimed: p150a power (no power or efficiency comparison); TensorRT or FP8 on the GPU; the compiled GPU path with
the p150a's exact host chain; N1.6's `gr1_long_text` layout. GPU numbers were not re-run for this card; they are copied with their
source lines from `ref-env-n16.md` §5 (which also records the machine state of each run).