# GR00T-N1.5-3B-p150 — Blackhole p150a vs RTX 5090 (same host, same weights, same observation)

Dates: GPU rows 2026-09-14, p150a rows 2026-09-14/15. Facts only: every number is copied from the file named in its
row. The two sides run **different code**: the GPU rows are NVIDIA's unmodified `Gr00tPolicy` (the official inference
path, eager PyTorch); the p150a rows are this package's tt-nn port (Stage-1: TTNN ops replayed from four Metal traces).
Read the caveats before comparing.

## What was run

| | RTX 5090 | Blackhole p150a |
|---|---|---|
| Code | NVIDIA [Isaac-GR00T](https://github.com/NVIDIA/Isaac-GR00T) tag `n1.5-release` (`4af2b622`), `Gr00tPolicy.get_action`, `FourierGr1ArmsOnlyDataConfig`, unmodified | `code/models/experimental/gr00t` (frozen snapshot `5dc0517ee3b`), `Gr00tTT` with `TTPolicy(dtype_policy="mixed_dit", trace_layout="per_stage")`, served by `code/gr00t_p150/server/app.py` |
| Weights | `nvidia/GR00T-N1.5-3B` @ `869830fc749c35f34771aa5209f923ac57e4564e`, bf16 shards from the HF cache | same snapshot; converted once into the device layout (bf16 everywhere, bfp8_b for the 16 DiT blocks' matmul weights) |
| Precision | official path: `torch.autocast(bfloat16)` + flash-attention 2 (`flash_attention_2` is hard-coded in the vendored Eagle2.5 config) | bf16 activations; HiFi2 matmul fidelity in the backbone/adapter, bfp8_b DiT weights; fp32 host pre/post-processing |
| Observation | the port's canonical golden sample: one 256×256 `ego_view` frame (GR1 sim `robot_sim.PickNPlace` traj 0 / step 100), GR1 state (7/7/6/6), instruction "pick the pear from the counter and place it in the plate", batch 1, fixed initial noise | identical inputs (`code/gr00t_p150/demo/n15/`); LLM sequence 296 tokens padded to the static 384 |
| GPU / host | NVIDIA GeForce RTX 5090 (sm_120), driver 580.126.18; settled at P1, 2985 MHz SM / 13,801 MHz mem, 190–250 W during the loop | one p150a (11×10 compute grid, 8 DRAM banks), same host, `/dev/tenstorrent/0` |
| Software | Python 3.10.20, torch 2.7.1+cu128, torchvision 0.22.1+cu128, transformers 4.51.3, flash-attn 2.8.3 (official prebuilt wheel; its kernels verified on sm_120 against SDPA to 1 bf16 ulp), cuDNN 9.7.1 | tt-metal main `668c2907575` (`v0.79.0-dev20260914-1`), torch 2.11.0+cpu; image: Python 3.12 |
| Loop | `time_n15.py`: 3 rounds × (20 warm-ups + 20 timed calls), `torch.cuda.synchronize()` around every call, GPU otherwise idle; **best median of the 3 rounds** (the round medians agree within 0.4 ms) | `benchmarks/bench_e2e.py`: 10 warm-ups + 50 timed calls, host wall clock (`perf_counter`), trace replays enqueued non-blocking with one synchronize per call; a second 50-call pass with a synchronize after every trace gives the per-trace split; **median** (p90 in brackets) |
| Source files | `docs/research/ref-env-n15.md` §6 (project research notes; timing JSON `golden/n15/timing.json`) — not shipped in this repo | `code/models/experimental/gr00t/benchmarks/results/e2e_stage1_n15.json` (shipped); served numbers from this image: `SERVING.md` §"Container validation" |

## Latency, batch 1 (ms)

### RTX 5090, official eager bf16 + flash-attn 2 path (`ref-env-n15.md` §6)

| stage | median ms |
|---|---:|
| `Gr00tPolicy.get_action` end-to-end (preprocess + model + un-normalise, includes a `deepcopy` of the observation) | **58.0** |
| preprocessing only (`apply_transforms`: crop/resize, PIL, tokenizer, image processor — CPU) | 2.17 |
| `model.get_action` (autocast bf16) | 55.4 |
| backbone `EagleBackbone.forward` (ViT + mlp1 + 12 × Qwen3 + lm_head) | 20.8 |
| — `extract_feature` (SigLIP ViT + mlp1) | 11.4 |
| — Qwen3 12 layers + lm_head (difference) | ≈ 9.4 |
| action head `get_action` | 34.6 |
| — `process_backbone_output` (vlln + 4 VL self-attention blocks) | 1.68 |
| — 4 × (action encoder + 16 DiT blocks + action decoder) (difference) | ≈ 32.9 ⇒ ≈ 8.2 per denoising step |

An fp32 (no autocast) model-only run measured 64–100 ms across runs and was not re-measured (not a target). No
`torch.compile` / TensorRT run was made for N1.5 (see caveats). NVIDIA's own N1.5 README quotes, for a single **H100**:
VLM backbone 23.18 ms, action head 4 × 6.18 = 24.7 ms, full model 47.88 ms.

### Blackhole p150a, Stage-1 trace path (`e2e_stage1_n15.json`, `stages.<name>.value` / `.p90`)

| stage | median ms (p90) |
|---|---:|
| **e2e** = host encode + upload + 4 traces + readback + host decode | **43.36** (44.08) |
| device = upload + traces + readback | 41.68 (42.25) |
| 4 trace replays, one synchronize | 37.83 (37.87) |
| — trace `vision` (SigLIP tower + projector) | 8.71 |
| — trace `llm` (Qwen3-1.7B[:12]) | 8.75 |
| — trace `adapter` (VL-LN + 4 VL-SA blocks + K/V hoist for the DiT cross-attention + state encoder) | 4.47 |
| — trace `denoise` (4 Euler steps × (action encoder + 16 DiT blocks + decoder)) | 16.01 |
| upload (host inputs → device buffers, incl. synchronize) | 3.77 (4.33) |
| readback of `action_pred` (D2H) | 0.09 |
| host encode (`Observation` → `ModelInputs`: image chain, prompt + tokenizer, sin/cos state) | 1.52 (1.82) |
| host decode (un-normalise) | 0.17 |
| e2e of the split pass (synchronize after every trace) | 43.92 (44.68) |

Same run re-executed on the frozen snapshot worktree before packaging: e2e 43.01, traces 37.82 (vision 8.70 · llm 8.73 ·
adapter 4.44 · denoise 15.99) — every trace median within 0.03 ms of the table above (`docs/publish/snapshot.md` §5.1).

Served over HTTP by this image (`tt-model serve` → `scripts/bench_http.py`, 10 warm-ups + 50 timed demo requests, quiet
host): **device 42.43 ms (p90 43.18) · server-side 45.40 ms (p90 49.20)**, client wall 47.6 ms on the same host —
`SERVING.md` §4 has the three boots, incl. two under a heavily loaded host (device 44.3–44.5, server-side 53.6–58.1 ms
medians) and the cold / warm boot times. The served device figure is 0.7 ms above `bench_e2e`'s device row (41.68 ms).

## Fidelity vs the fp32 reference (same observation, same initial noise)

The fp32 reference is NVIDIA's own `Gr00tPolicy` run on CPU in fp32 (math SDPA). Both bf16 paths are measured
against it; PCC in float64 over the flattened tensor.

| tensor | NVIDIA bf16 + FA2 on the RTX 5090 (`ref-env-n15.md` §5.1) | p150a Stage-1 path (`tests/tt/results/test_e2e_untraced_taps_vs_golden_n15_20260914-230740.json`) | gate applied to the p150a (multi-sample rule: twice the worst deviation of NVIDIA's bf16 path over 9 observations) |
|---|---:|---:|---:|
| `backbone_features` (post final RMSNorm) | 0.999006 | 0.995863 | 0.995 |
| `vl_self_attention_out` (= DiT `encoder_hidden_states`) | 0.997315 | 0.991189 | 0.991 |
| `vit_post_layernorm_out` | 0.999000 | 0.995057 | 0.993 |
| `action_pred_normalized` (1×16×32) | 0.999991 | 0.999980 (max\|d\| 0.0143) | 0.9999 (+ abs 0.027) |
| `unnormalized_action.left_arm` / `right_arm` / `right_hand` | 0.99998 / 0.99998 / 0.99999 | 0.999947 / 0.999972 / 0.999994 (max\|d\| 0.021 / 0.017 / 0.029 rad) | 0.9999 (+ abs 0.050 / 0.038 / 0.058) |
| `unnormalized_action.left_hand` (range 0.099 rad in this sample — PCC not meaningful) | 0.9525 | 0.943352 (max\|d\| 0.018 rad) | abs 0.046 only |

Over the multi-sample set (9 observations, `golden-multisample.md` §4.1) NVIDIA's bf16 path itself drops to
`backbone_features` 0.99771, `vl_self_attention_out` 0.99573 and `action_pred_normalized` 0.999987; the p150a is held
to gates derived from those minima. All 58 rows of the p150a run pass; the final action chunk agrees with the fp32
reference to PCC ≥ 0.99995 on every gated group.

## Comparison with matching definitions

Ratio = RTX 5090 ms / p150a ms (> 1 means the p150a is faster).

| row | RTX 5090 (official eager bf16 + FA2) | p150a (Stage-1 traces) | ratio |
|---|---:|---:|---:|
| end-to-end: `Gr00tPolicy.get_action` vs port e2e (both: host preprocessing + model + un-normalisation, batch 1) | 58.0 | 43.36 | **1.34** |
| model only: `model.get_action` vs port device (upload + traces + readback) | 55.4 | 41.68 | **1.33** |
| backbone: `EagleBackbone.forward` vs traces `vision` + `llm` | 20.8 | 17.46 | 1.19 |
| action head: `ActionHead.get_action` (VL-SA + 4 denoising steps) vs traces `adapter` + `denoise` | 34.6 | 20.48 | 1.69 |
| host preprocessing: `apply_transforms` vs port encode | 2.17 | 1.52 | 1.43 |

Reading: at batch 1 the eager GPU path is kernel-launch bound (the action head is 60 % of its time at 1/3 of the FLOPs,
≈ 16 blocks × ~15 kernels per denoising step); the p150a's traces replay the same graph with no launch overhead, which is
where the 1.7× on the head comes from, while the two backbones are within 20 %.

## Caveats — read before quoting a ratio

1. **Eager GPU only.** No `torch.compile` or TensorRT run was made for N1.5. For N1.6 on the same GPU,
   `torch.compile(DiT.forward, mode="max-autotune")` cut the official e2e from 63.1 ms to 33.1 ms (`ref-env-n16.md` §5)
   and NVIDIA's N1.6 README lists 37 ms compiled on an RTX 5090; a compiled N1.5 would very likely be faster than this
   Stage-1 p150a path. The p150a's own next step (the persistent-megakernel denoise, Stage 2) is not in this image either.
2. **Different code, similar precision class.** GPU: bf16 autocast + FA2 (NVIDIA's shipped path). p150a: bf16 with
   bfp8_b DiT weights and HiFi2 matmul fidelity. Both are compared against the same fp32 reference above; neither is fp32.
3. **Stage boundaries differ slightly.** The GPU "backbone" includes `lm_head`; the p150a `adapter` trace holds the VL
   self-attention (GPU: inside the action head) plus the K/V hoist and state encoder. Only the e2e and model rows are
   like-for-like.
4. **Loops differ** (3 × 20 timed calls best-median vs 50 timed calls median; `cuda.synchronize` per call vs one
   host-side synchronize per call). Both exclude HTTP/JSON framing. The GPU e2e includes a `deepcopy` of the observation
   (NVIDIA's code); the p150a e2e includes the 3.8 ms input upload.
5. **Power / efficiency: not compared.** GPU draw during the loop was 190–250 W (idle not recorded in this run); p150a
   power was not measured.
6. **Batch 1, one sample.** Timings are for the canonical observation; the port's traces are shape-locked, so any
   GR1 request with ≤ 102 text tokens takes the same device time (the sequence is padded to 384).

## Reproduce

```bash
# p150a, served (any host with the package pulled):
tt serve changh95/GR00T-N1.5-3B-p150
python3 code/scripts/bench_http.py --url http://127.0.0.1:20000 --reps 50 --warmup 10      # server-side timing_ms + client wall
# p150a, the port benchmark (needs a built tt-metal 668c2907575 tree, the pinned weights and the project's golden observation):
python code/models/experimental/gr00t/benchmarks/bench_e2e.py --version n15 --out <dir>
# RTX 5090: Isaac-GR00T n1.5-release + the environment of ref-env-n15.md §1.2; the timing script (tt_golden/time_n15.py)
# lives in the project's research tree, not in this repo.
```