Stage-2 release: DiT denoise as one persistent megakernel (snapshot 009de892, image dfb9c3889ab1)

Replaces the Stage-1 (TTNN-op denoise) package with the megakernel release: image sha256:dfb9c3889ab17cc1d8187085b27e854d0311fb974d729efe88db9bfdce51869f (tt-model/gr00t-n1.5-3b-p150:dfb9c3889ab1), port snapshot 009de892c895 on tt-metal 668c290757, code/ digest d87906471090789d. `tt-model push --publish` uploaded code/, image/, tt_kernel_manifest.json, requirements.lock and the generated card and pruned the code/ and image/ files this bundle no longer ships; this commit restores the reviewed staging card (front matter: license other / nvidia-license, pipeline_tag robotics, base_model nvidia/GR00T-N1.5-3B, 13 tags; body = the tt-model.yaml card with the simplified quickstart bullets) and the authored files tt-model.yaml, SERVING.md, GPU_COMPARISON.md, LICENSE, LICENSE-NOTICE.md, requirements.lock and media/*. Card text per docs/publish/review-mk.md.

GPU_COMPARISON.md

@@ -1,22 +1,23 @@
 # GR00T-N1.5-3B-p150 — Blackhole p150a vs RTX 5090 (same host, same weights, same observation)
 
-Dates: GPU rows 2026-09-14, p150a rows 2026-09-14/15. Facts only: every number is copied from the file named in its
-row. The two sides run **different code**: the GPU rows are NVIDIA's unmodified `Gr00tPolicy` (the official inference
-path, eager PyTorch); the p150a rows are this package's tt-nn port (Stage-1: TTNN ops replayed from four Metal traces).
-Read the caveats before comparing.
+Dates: GPU rows 2026-09-14, p150a rows 2026-09-18 (megakernel release; the Stage-1 rows of the 2026-09-14 release are kept
+as the reference). Facts only: every number is copied from the file named in its row. The two sides run **different
+code**: the GPU rows are NVIDIA's unmodified `Gr00tPolicy` (the official inference path, eager PyTorch); the p150a rows
+are this package's tt-nn port (Stage 2: vision / LLM / adapter as TTNN ops replayed from three Metal traces, the DiT
+denoise as one persistent megakernel `generic_op` in the fourth). Read the caveats before comparing.
 
 ## What was run
 
 | | RTX 5090 | Blackhole p150a |
 |---|---|---|
-| Code | NVIDIA [Isaac-GR00T](https://github.com/NVIDIA/Isaac-GR00T) tag `n1.5-release` (`4af2b622`), `Gr00tPolicy.get_action`, `FourierGr1ArmsOnlyDataConfig`, unmodified | `code/models/experimental/gr00t` (frozen snapshot `5dc0517ee3b`), `Gr00tTT` with `TTPolicy(dtype_policy="mixed_dit", trace_layout="per_stage")`, served by `code/gr00t_p150/server/app.py` |
-| Weights | `nvidia/GR00T-N1.5-3B` @ `869830fc749c35f34771aa5209f923ac57e4564e`, bf16 shards from the HF cache | same snapshot; converted once into the device layout (bf16 everywhere, bfp8_b for the 16 DiT blocks' matmul weights) |
-| Precision | official path: `torch.autocast(bfloat16)` + flash-attention 2 (`flash_attention_2` is hard-coded in the vendored Eagle2.5 config) | bf16 activations; HiFi2 matmul fidelity in the backbone/adapter, bfp8_b DiT weights; fp32 host pre/post-processing |
+| Code | NVIDIA [Isaac-GR00T](https://github.com/NVIDIA/Isaac-GR00T) tag `n1.5-release` (`4af2b622`), `Gr00tPolicy.get_action`, `FourierGr1ArmsOnlyDataConfig`, unmodified | `code/models/experimental/gr00t` (frozen snapshot `009de892c895`), `Gr00tTT` with `TTPolicy(dtype_policy="mixed_dit", dit_backend="megakernel", mk_arena_dtype="bfp8_b", trace_layout="per_stage")`, device opened by `open_model_device(num_command_queues=2)`, served by `code/gr00t_p150/server/app.py`; Stage-1 reference rows: the same tree with `dit_backend="ttnn"` |
+| Weights | `nvidia/GR00T-N1.5-3B` @ `869830fc749c35f34771aa5209f923ac57e4564e`, bf16 shards from the HF cache | same snapshot; converted once into the device layout (bf16 everywhere; the 16 DiT blocks' matmul weights bfp8_b, packed into the megakernel's DRAM arena — 496 MB streamed per denoising step) |
+| Precision | official path: `torch.autocast(bfloat16)` + flash-attention 2 (`flash_attention_2` is hard-coded in the vendored Eagle2.5 config) | bf16 activations; HiFi2 matmul fidelity in the backbone / adapter and inside the megakernel's blocks (bf16 / HiFi4 tail ops), bfp8_b DiT weights; fp32 host pre/post-processing |
 | Observation | the port's canonical golden sample: one 256×256 `ego_view` frame (GR1 sim `robot_sim.PickNPlace` traj 0 / step 100), GR1 state (7/7/6/6), instruction "pick the pear from the counter and place it in the plate", batch 1, fixed initial noise | identical inputs (`code/gr00t_p150/demo/n15/`); LLM sequence 296 tokens padded to the static 384 |
-| GPU / host | NVIDIA GeForce RTX 5090 (sm_120), driver 580.126.18; settled at P1, 2985 MHz SM / 13,801 MHz mem, 190–250 W during the loop | one p150a (11×10 compute grid, 8 DRAM banks), same host, `/dev/tenstorrent/0` |
+| GPU / host | NVIDIA GeForce RTX 5090 (sm_120), driver 580.126.18; settled at P1, 2985 MHz SM / 13,801 MHz mem, 190–250 W during the loop | one p150a (11×10 compute grid, 8 DRAM banks; the megakernel uses 110 cores), same host, `/dev/tenstorrent/0` |
 | Software | Python 3.10.20, torch 2.7.1+cu128, torchvision 0.22.1+cu128, transformers 4.51.3, flash-attn 2.8.3 (official prebuilt wheel; its kernels verified on sm_120 against SDPA to 1 bf16 ulp), cuDNN 9.7.1 | tt-metal main `668c2907575` (`v0.79.0-dev20260914-1`), torch 2.11.0+cpu; image: Python 3.12 |
-| Loop | `time_n15.py`: 3 rounds × (20 warm-ups + 20 timed calls), `torch.cuda.synchronize()` around every call, GPU otherwise idle; **best median of the 3 rounds** (the round medians agree within 0.4 ms) | `benchmarks/bench_e2e.py`: 10 warm-ups + 50 timed calls, host wall clock (`perf_counter`), trace replays enqueued non-blocking with one synchronize per call; a second 50-call pass with a synchronize after every trace gives the per-trace split; **median** (p90 in brackets) |
-| Source files | `docs/research/ref-env-n15.md` §6 (project research notes; timing JSON `golden/n15/timing.json`) — not shipped in this repo | `code/models/experimental/gr00t/benchmarks/results/e2e_stage1_n15.json` (shipped); served numbers from this image: `SERVING.md` §"Container validation" |
+| Loop | `time_n15.py`: 3 rounds × (20 warm-ups + 20 timed calls), `torch.cuda.synchronize()` around every call, GPU otherwise idle; **best median of the 3 rounds** (the round medians agree within 0.4 ms) | `benchmarks/bench_e2e.py`: 10 warm-ups + 50 timed calls, host wall clock (`perf_counter`), input writes on CQ 1, trace replays enqueued non-blocking with one synchronize per call; a second 50-call pass with a synchronize after every trace gives the per-trace split; **median** (p90 in brackets) |
+| Source files | `docs/research/ref-env-n15.md` §6 (project research notes; timing JSON `golden/n15/timing.json`) — not shipped in this repo | `code/models/experimental/gr00t/benchmarks/results/bench_e2e_n15_20260918-050550.json` (megakernel, the frozen snapshot), `bench_e2e_n15_20260918-032243.json` (same tree, `--dit-backend ttnn`), `e2e_stage1_n15.json` (the 2026-09-14 Stage-1 release) — all shipped; served numbers from this image: `SERVING.md` §4 |
 
 ## Latency, batch 1 (ms)
 
@@ -38,84 +39,102 @@ An fp32 (no autocast) model-only run measured 64–100 ms across runs and was no
 `torch.compile` / TensorRT run was made for N1.5 (see caveats). NVIDIA's own N1.5 README quotes, for a single **H100**:
 VLM backbone 23.18 ms, action head 4 × 6.18 = 24.7 ms, full model 47.88 ms.
 
-### Blackhole p150a, Stage-1 trace path (`e2e_stage1_n15.json`, `stages.<name>.value` / `.p90`)
+### Blackhole p150a (`stages.<name>.value` / `.p90` of the named JSONs)
 
-| stage | median ms (p90) |
-|---|---:|
-| **e2e** = host encode + upload + 4 traces + readback + host decode | **43.36** (44.08) |
-| device = upload + traces + readback | 41.68 (42.25) |
-| 4 trace replays, one synchronize | 37.83 (37.87) |
-| — trace `vision` (SigLIP tower + projector) | 8.71 |
-| — trace `llm` (Qwen3-1.7B[:12]) | 8.75 |
-| — trace `adapter` (VL-LN + 4 VL-SA blocks + K/V hoist for the DiT cross-attention + state encoder) | 4.47 |
-| — trace `denoise` (4 Euler steps × (action encoder + 16 DiT blocks + decoder)) | 16.01 |
-| upload (host inputs → device buffers, incl. synchronize) | 3.77 (4.33) |
-| readback of `action_pred` (D2H) | 0.09 |
-| host encode (`Observation` → `ModelInputs`: image chain, prompt + tokenizer, sin/cos state) | 1.52 (1.82) |
-| host decode (un-normalise) | 0.17 |
-| e2e of the split pass (synchronize after every trace) | 43.92 (44.68) |
-
-Same run re-executed on the frozen snapshot worktree before packaging: e2e 43.01, traces 37.82 (vision 8.70 · llm 8.73 ·
-adapter 4.44 · denoise 15.99) — every trace median within 0.03 ms of the table above (`docs/publish/snapshot.md` §5.1).
-
-Served over HTTP by this image (`tt-model serve` → `scripts/bench_http.py`, 10 warm-ups + 50 timed demo requests, quiet
-host): **device 42.43 ms (p90 43.18) · server-side 45.40 ms (p90 49.20)**, client wall 47.6 ms on the same host —
-`SERVING.md` §4 has the three boots, incl. two under a heavily loaded host (device 44.3–44.5, server-side 53.6–58.1 ms
-medians) and the cold / warm boot times. The served device figure is 0.7 ms above `bench_e2e`'s device row (41.68 ms).
+| stage | megakernel denoise (this release; `bench_e2e_n15_20260918-050550.json`) | same tree, Stage-1 TTNN denoise (`bench_e2e_n15_20260918-032243.json`) | Stage-1 release of 2026-09-14 (`e2e_stage1_n15.json`) |
+|---|---:|---:|---:|
+| **e2e** = host encode + upload + 4 traces + readback + host decode | **33.59** (33.91) | 40.38 (40.76) | 43.36 (44.08) |
+| device = upload + traces + readback | 32.41 (32.61) | 39.14 (39.39) | 41.68 (42.25) |
+| 4 trace replays, one synchronize | 31.12 (31.15) | 37.87 (37.91) | 37.83 (37.87) |
+| — trace `vision` (SigLIP tower + projector) | 8.73 | 8.73 | 8.71 |
+| — trace `llm` (Qwen3-1.7B[:12]) | 8.78 | 8.76 | 8.75 |
+| — trace `adapter` (VL-LN + 4 VL-SA blocks + K/V hoist for the DiT cross-attention + state encoder) | 4.86 (intermediates in DRAM) | 4.48 | 4.47 |
+| — trace `denoise` (4 Euler steps × (action encoder + 16 DiT blocks + decoder)) | **8.93** = 2.23 per step (one `generic_op`) | 16.02 | 16.01 |
+| upload (host inputs → device buffers; CQ 1 in the 09-18 rows) | 1.25 (1.44) | 1.24 (1.41) | 3.77 (4.33) |
+| readback of `action_pred` (D2H) | 0.05 | 0.07 | 0.09 |
+| host encode (`Observation` → `ModelInputs`: image chain, prompt + tokenizer, sin/cos state) | 0.99 (1.22) | 1.05 (1.21) | 1.52 (1.82) |
+| host decode (un-normalise) | 0.13 | 0.16 | 0.17 |
+| e2e of the split pass (synchronize after every trace) | 34.16 (34.89) | 40.64 (40.96) | 43.92 (44.68) |
+
+The megakernel column is the frozen snapshot's own run (`commit 009de892c895`); the same benchmark on the live tree at the
+K4b checkpoint gave e2e 33.69 / device 32.48 / denoise 8.92 ms (`bench_e2e_n15_20260918-043024.json`, shipped) — every
+device-side row within 0.1 ms. Against the same-tree TTNN denoise the megakernel saves **6.8 ms end-to-end (−16.8 %)**, all
+of it in the denoise trace (16.02 → 8.93 ms, **−44 %**; per step 4.00 → 2.23 ms) minus the adapter's DRAM placement
+(+0.38 ms). The megakernel alone, traced and replayed 20 times, runs the four steps in 8.86 ms
+(`bench_mk_step_n15_20260918-035610.json`: 2.216 ms per step; the bfp8 streaming floor at the measured 414 GB/s would be
+1.20 ms per step, so the kernel is 1.85× off its bandwidth floor — the block is compute- and gather-bound, not stream-bound).
+The upload / host-encode differences between the 09-18 and 09-14 columns come from the port's 2-CQ input path and
+device-side embedding gather added between the releases, not from the megakernel.
+
+Served over HTTP by this image (`tt-model serve` → `scripts/bench_http.py`, 10 warm-ups + 50 timed demo requests, warm
+boot): **device 32.5 ms (p90 32.82) · server-side 34.6 ms (p90 35.39)**,
+client wall 36.3 ms on the same host — `SERVING.md` §4 has both boots and the cold / warm boot times. The Stage-1
+image of 2026-09-14 served 42.43 ms device / 45.40 ms server-side on the same host.
 
 ## Fidelity vs the fp32 reference (same observation, same initial noise)
 
 The fp32 reference is NVIDIA's own `Gr00tPolicy` run on CPU in fp32 (math SDPA). Both bf16 paths are measured
 against it; PCC in float64 over the flattened tensor.
 
-| tensor | NVIDIA bf16 + FA2 on the RTX 5090 (`ref-env-n15.md` §5.1) | p150a Stage-1 path (`tests/tt/results/test_e2e_untraced_taps_vs_golden_n15_20260914-230740.json`; that run records the port's `two` trace layout — the `per_stage` re-run on the frozen snapshot gives identical values, `SERVING.md` §2) | gate applied to the p150a (multi-sample rule: twice the worst deviation of NVIDIA's bf16 path over 9 observations) |
-|---|---:|---:|---:|
-| `backbone_features` (post final RMSNorm) | 0.999006 | 0.995863 | 0.995 |
-| `vl_self_attention_out` (= DiT `encoder_hidden_states`) | 0.997315 | 0.991189 | 0.991 |
-| `vit_post_layernorm_out` | 0.999000 | 0.995057 | 0.993 |
-| `action_pred_normalized` (1×16×32) | 0.999991 | 0.999980 (max\|d\| 0.0143) | 0.9999 (+ abs 0.027) |
-| `unnormalized_action.left_arm` / `right_arm` / `right_hand` | 0.99998 / 0.99998 / 0.99999 | 0.999947 / 0.999972 / 0.999994 (max\|d\| 0.021 / 0.017 / 0.029 rad) | 0.9999 (+ abs 0.050 / 0.038 / 0.058) |
-| `unnormalized_action.left_hand` (range 0.099 rad in this sample — PCC not meaningful) | 0.9525 | 0.943352 (max\|d\| 0.018 rad) | abs 0.046 only |
-
-Over the multi-sample set (9 observations, `golden-multisample.md` §4.1) NVIDIA's bf16 path itself drops to
-`backbone_features` 0.99771, `vl_self_attention_out` 0.99573 and `action_pred_normalized` 0.999987; the p150a is held
-to gates derived from those minima. All 58 rows of the p150a run pass; the final action chunk agrees with the fp32
-reference to PCC ≥ 0.99995 on every gated group.
+| tensor | NVIDIA bf16 + FA2 on the RTX 5090 (`ref-env-n15.md` §5.1) | p150a, megakernel denoise (`tests/tt/results/test_e2e_untraced_taps_vs_golden_n15_20260918-044858.json`, frozen snapshot) | p150a, Stage-1 release of 2026-09-14 (`test_e2e_untraced_taps_vs_golden_n15_20260914-230740.json` of the port project's `build-n15.md` §6 — JSON not shipped in this repo) | gate applied to the p150a (multi-sample rule: twice the worst deviation of NVIDIA's bf16 path over 9 observations) |
+|---|---:|---:|---:|---:|
+| `backbone_features` (post final RMSNorm) | 0.999006 | 0.995863 | 0.995863 | 0.995 |
+| `vl_self_attention_out` (= DiT `encoder_hidden_states`) | 0.997315 | 0.991189 | 0.991189 | 0.991 |
+| `vit_post_layernorm_out` | 0.999000 | 0.995057 | 0.995057 | 0.993 |
+| `action_pred_normalized` (1×16×32) | 0.999991 | 0.999980 (max\|d\| 0.0137) | 0.999980 (max\|d\| 0.0143) | 0.9999 (+ abs 0.027) |
+| `unnormalized_action.left_arm` / `right_arm` / `right_hand` | 0.99998 / 0.99998 / 0.99999 | 0.999944 / 0.999971 / 0.999993 (max\|d\| 0.023 / 0.017 / 0.029 rad) | 0.999947 / 0.999972 / 0.999994 (max\|d\| 0.021 / 0.017 / 0.029 rad) | 0.9999 (+ abs 0.050 / 0.038 / 0.058) |
+| `unnormalized_action.left_hand` (range 0.099 rad in this sample — PCC not meaningful) | 0.9525 | 0.928345 (max\|d\| 0.030 rad) | 0.943352 (max\|d\| 0.018 rad) | abs 0.046 only |
+
+The backbone and adapter rows are bit-identical between the two p150a columns (the megakernel does not touch them); the
+action rows move by ≤ 3e-6 PCC. Megakernel vs the TTNN denoise on the same device and observation: `action_pred_normalized`
+PCC 0.999997, per-step velocities ≥ 0.99998 (`test_mk_e2e_backends_agree_canonical_n15_20260918-042408.json`). Over the
+multi-sample set (9 observations, `golden-multisample.md` §4.1) NVIDIA's bf16 path itself drops to `backbone_features`
+0.99771, `vl_self_attention_out` 0.99573 and `action_pred_normalized` 0.999987; the p150a is held to gates derived from
+those minima. All 30 rows of the megakernel run pass (29 gated rows incl. the `left_hand` abs row + the two-runs-bit-equal
+row; the Stage-1 path had 58 rows because its per-step DiT intermediates are observable and gated — inside one
+`generic_op` they are not). On the 9 multi-sample observations the megakernel's gate outcomes equal the TTNN path's on 8;
+the exception is `right_arm` on `t3_s300_seed2` at 0.999869 vs 0.9999 (`SERVING.md` §7).
 
 ## Comparison with matching definitions
 
 Ratio = RTX 5090 ms / p150a ms (> 1 means the p150a is faster).
 
-| row | RTX 5090 (official eager bf16 + FA2) | p150a (Stage-1 traces) | ratio |
-|---|---:|---:|---:|
-| end-to-end: `Gr00tPolicy.get_action` vs port e2e (both: host preprocessing + model + un-normalisation, batch 1) | 58.0 | 43.36 | **1.34** |
-| model only: `model.get_action` vs port device (upload + traces + readback) | 55.4 | 41.68 | **1.33** |
-| backbone: `EagleBackbone.forward` vs traces `vision` + `llm` | 20.8 | 17.46 | 1.19 |
-| action head: `ActionHead.get_action` (VL-SA + 4 denoising steps) vs traces `adapter` + `denoise` | 34.6 | 20.48 | 1.69 |
-| host preprocessing: `apply_transforms` vs port encode | 2.17 | 1.52 | 1.43 |
+| row | RTX 5090 (official eager bf16 + FA2) | p150a, megakernel denoise | ratio | p150a, Stage-1 release (2026-09-14) | ratio |
+|---|---:|---:|---:|---:|---:|
+| end-to-end: `Gr00tPolicy.get_action` vs port e2e (both: host preprocessing + model + un-normalisation, batch 1) | 58.0 | 33.59 | **1.73** | 43.36 | 1.34 |
+| model only: `model.get_action` vs port device (upload + traces + readback) | 55.4 | 32.41 | **1.71** | 41.68 | 1.33 |
+| backbone: `EagleBackbone.forward` vs traces `vision` + `llm` | 20.8 | 17.51 | 1.19 | 17.46 | 1.19 |
+| action head: `ActionHead.get_action` (VL-SA + 4 denoising steps) vs traces `adapter` + `denoise` | 34.6 | 13.79 | **2.51** | 20.48 | 1.69 |
+| host preprocessing: `apply_transforms` vs port encode | 2.17 | 0.99 | 2.19 | 1.52 | 1.43 |
 
 Reading: at batch 1 the eager GPU path is kernel-launch bound (the action head is 60 % of its time at 1/3 of the FLOPs,
-≈ 16 blocks × ~15 kernels per denoising step); the p150a's traces replay the same graph with no launch overhead, which is
-where the 1.7× on the head comes from, while the two backbones are within 20 %.
+≈ 16 blocks × ~15 kernels per denoising step); the p150a replays the backbone graph with no launch overhead and runs the
+whole head as one program whose weights stream from DRAM once per step, which is where the 2.5× on the head comes from,
+while the two backbones are within 20 %. The backbone (vision + LLM, 17.5 ms) is now the largest block of the p150a's
+33.6 ms; the denoise is 27 %.
 
 ## Caveats — read before quoting a ratio
 
 1. **Eager GPU only.** No `torch.compile` or TensorRT run was made for N1.5. For N1.6 on the same GPU,
    `torch.compile(DiT.forward, mode="max-autotune")` cut the official e2e from 63.1 ms to 33.1 ms (`ref-env-n16.md` §5)
-   and NVIDIA's N1.6 README lists 37 ms compiled on an RTX 5090; a compiled N1.5 would very likely be faster than this
-   Stage-1 p150a path. The p150a's own next step (the persistent-megakernel denoise, Stage 2) is not in this image either.
+   and NVIDIA's N1.6 README lists 37 ms compiled on an RTX 5090; a compiled N1.5 would very likely land near this
+   p150a figure rather than 1.7× behind it.
 2. **Different code, similar precision class.** GPU: bf16 autocast + FA2 (NVIDIA's shipped path). p150a: bf16 with
-   bfp8_b DiT weights and HiFi2 matmul fidelity. Both are compared against the same fp32 reference above; neither is fp32.
+   bfp8_b DiT weights (streamed by the megakernel) and HiFi2 matmul fidelity. Both are compared against the same fp32
+   reference above; neither is fp32.
 3. **Stage boundaries differ slightly.** The GPU "backbone" includes `lm_head`; the p150a `adapter` trace holds the VL
    self-attention (GPU: inside the action head) plus the K/V hoist and state encoder. Only the e2e and model rows are
    like-for-like.
 4. **Loops differ** (3 × 20 timed calls best-median vs 50 timed calls median; `cuda.synchronize` per call vs one
    host-side synchronize per call). Both exclude HTTP/JSON framing. The GPU e2e includes a `deepcopy` of the observation
-   (NVIDIA's code); the p150a e2e includes the 3.8 ms input upload.
+   (NVIDIA's code); the p150a e2e includes the 1.2 ms input upload.
 5. **Power / efficiency: not compared.** GPU draw during the loop was 190–250 W (idle not recorded in this run); p150a
    power was not measured.
 6. **Batch 1, one sample.** Timings are for the canonical observation; the port's traces are shape-locked, so any
    GR1 request with ≤ 102 text tokens takes the same device time (the sequence is padded to 384).
+7. **Megakernel disclosures.** The p150a rows of this release run the denoise with 64 KiB less allocatable L1 per core
+   and the adapter's intermediates in DRAM, expose no DiT intermediates for gating, and miss one multi-sample action gate
+   the Stage-1 path passes (`SERVING.md` §7). `GR00T_DIT_BACKEND=ttnn` reproduces the Stage-1 rows on the same image.
 
 ## Reproduce
 
@@ -124,7 +143,8 @@ where the 1.7× on the head comes from, while the two backbones are within 20 %.
 tt serve changh95/GR00T-N1.5-3B-p150
 python3 code/scripts/bench_http.py --url http://127.0.0.1:20000 --reps 50 --warmup 10      # server-side timing_ms + client wall
 # p150a, the port benchmark (needs a built tt-metal 668c2907575 tree, the pinned weights and the project's golden observation):
-python code/models/experimental/gr00t/benchmarks/bench_e2e.py --version n15 --out <dir>
+python code/models/experimental/gr00t/benchmarks/bench_e2e.py --version n15 --out <dir>                      # megakernel (default)
+python code/models/experimental/gr00t/benchmarks/bench_e2e.py --version n15 --dit-backend ttnn --out <dir>   # Stage-1 reference
 # RTX 5090: Isaac-GR00T n1.5-release + the environment of ref-env-n15.md §1.2; the timing script (tt_golden/time_n15.py)
 # lives in the project's research tree, not in this repo.
 ```

LICENSE-NOTICE.md

@@ -8,7 +8,7 @@ two licences; nothing here grants more than its source does.
 | Model weights `nvidia/GR00T-N1.5-3B` @ `869830fc749c35f34771aa5209f923ac57e4564e` | **not included** — `tt-model serve` / `code/scripts/download_weights.sh` fetch the pinned snapshot into your HF cache | **NVIDIA License** (copy in [`LICENSE`](LICENSE); original: [`LICENSE` in the weights repo](https://huggingface.co/nvidia/GR00T-N1.5-3B/blob/main/LICENSE)) — use limited to research or evaluation (§3.3), redistribution only under the same licence with a copy included (§3.1), derivative works carry the same use limitation (§3.2) | NVIDIA |
 | Qwen2 tokenizer files (`vocab.json`, `merges.txt`, `tokenizer_config.json`, `special_tokens_map.json`, `added_tokens.json`) | `code/gr00t_p150/assets/tokenizer/n15/` | Apache-2.0 (copied from `gr00t/model/backbone/eagle2_hg_model/` of [NVIDIA/Isaac-GR00T](https://github.com/NVIDIA/Isaac-GR00T) at tag `n1.5-release`, an Apache-2.0 repository) | NVIDIA / Alibaba Qwen |
 | Demo frame and joint state (`media/demo_ego_view.png`, `media/demo_observation.json`, `code/gr00t_p150/demo/n15/`) | `media/`, `code/gr00t_p150/demo/n15/` | one step of the GR1 `robot_sim.PickNPlace` demo data shipped with Isaac-GR00T (`demo_data/`, Apache-2.0 repository); the reference actions in `expected.json` were computed from those inputs with the weights above | NVIDIA |
-| The tt-nn port (`code/models/experimental/gr00t/`), the policy server (`code/gr00t_p150/`), the scripts (`code/scripts/`) and the tt-metal files copied into `code/models/{common,tt_dit,demos}/` | `code/` | **Apache-2.0** (SPDX headers; © 2026 Tenstorrent). Because the port's device-layout weights and the served outputs are derivative works of the NVIDIA weights, running this package is bound by the NVIDIA License's non-commercial use limitation (§3.2/§3.3) | Tenstorrent |
+| The tt-nn port (`code/models/experimental/gr00t/`), the policy server (`code/gr00t_p150/`), the scripts (`code/scripts/`) and the tt-metal files copied into `code/models/{common,tt_dit,demos}/` (incl. the four `deepseek_v3_b1/unified_kernels/*.hpp` device headers the megakernel includes) | `code/` | **Apache-2.0** (SPDX headers; © 2026 Tenstorrent). Because the port's device-layout weights and the served outputs are derivative works of the NVIDIA weights, running this package is bound by the NVIDIA License's non-commercial use limitation (§3.2/§3.3) | Tenstorrent |
 | The container image (`image/`) | `image/` | tt-metal / tt-nn (Apache-2.0), Ubuntu 22.04 base and the Python packages listed in `requirements.lock` under their own licences; **contains no weights** | Tenstorrent, upstream projects |
 
 Model card front matter therefore declares `license: other`, `license_name: nvidia-license`,

README.md

@@ -4,11 +4,24 @@ tags:
 - p150
 - tt-dit-server
 - tt-model-cache
-- tt-model-catalog
 - tt-model-container
+- tenstorrent
+- ttnn
+- tt-metal
+- tt-nn
+- robotics
+- vla
+- gr00t
+- tt-model-catalog
+pipeline_tag: robotics
+license: other
+license_name: nvidia-license
+license_link: https://huggingface.co/nvidia/GR00T-N1.5-3B/blob/main/LICENSE
+base_model:
+- nvidia/GR00T-N1.5-3B
 ---
 
-# gr00t-n1.5-3b-p150
+# GR00T-N1.5-3B-p150
 
 NVIDIA Isaac GR00T N1.5 (3B vision-language-action policy) running on one Tenstorrent Blackhole p150a via tt-nn: one camera frame, proprioceptive state and a task instruction in, a 16-step chunk of GR1 arm and hand joint targets out. Stage-2 release: vision tower, LLM and VL adapter as TTNN ops replayed from Metal traces (bf16); the whole DiT denoise (4 steps × 16 blocks) as **one persistent megakernel** on 110 Tensix cores, streaming its bfp8 weights from a DRAM arena once per step. `GR00T_DIT_BACKEND=ttnn` selects the Stage-1 TTNN denoise.
 Weights: [nvidia/GR00T-N1.5-3B](https://huggingface.co/nvidia/GR00T-N1.5-3B) · Paper: [arXiv:2503.14734](https://arxiv.org/abs/2503.14734) · Upstream code: [NVIDIA/Isaac-GR00T](https://github.com/NVIDIA/Isaac-GR00T) (`n1.5-release`) · Port: `code/models/experimental/gr00t` and the server `code/gr00t_p150` in this repo
@@ -24,7 +37,8 @@ tt-model pull  changh95/GR00T-N1.5-3B-p150 --with-weights
 tt-model serve changh95/GR00T-N1.5-3B-p150
 ```
 
-`pull --with-weights` downloads the Docker image and the [`nvidia/GR00T-N1.5-3B`](https://huggingface.co/nvidia/GR00T-N1.5-3B) weights at `869830fc749c35f34771aa5209f923ac57e4564e` (into your HF cache; they are not in the image). `serve` starts the model's own HTTP server on port 20000 (or the next free port, if that one is busy); the first start compiles kernels for your device, which takes several minutes, and the server is ready when it logs `Application startup complete`.
+- Weights [`nvidia/GR00T-N1.5-3B`](https://huggingface.co/nvidia/GR00T-N1.5-3B) at `869830fc749c` (5.45 GB) go to your HF cache; the image does not contain them.
+- Serves on port 20000 (or the next free port); ready when the log says `Application startup complete`.
 
 ### Run with tt-cli
 
@@ -100,4 +114,3 @@ The exact sources the image was built from — `code/` in this repo is byte-iden
 | tt-metal | [`668c290757550588d0ce46b180c344a462a2aaf5`](https://github.com/tenstorrent/tt-metal/commit/668c290757550588d0ce46b180c344a462a2aaf5) |
 | `code/` digest | `d87906471090789d` (sha256, first 16 hex digits) |
 | built | 2026-09-17T20:23:50+00:00 by tt-model 0.1.0 |
-

SERVING.md

@@ -3,17 +3,20 @@
 This repo is a **tt-model container package source**: `tt-model.yaml` + the code under `code/`. `tt-model package
 --container` builds the OCI image (tt-metal `668c2907575` built from source + the port + the HTTP stack),
 `tt-model serve` runs it against the card, `tt-model push` publishes it to this HF repo. The server is
-`code/gr00t_p150/server/app.py` (FastAPI, `kind: tt-dit-server`); what it runs on the device is the port's **Stage-1**
-path — every op a TTNN op, captured once into four Metal traces (`vision`, `llm`, `adapter`, `denoise`) and replayed per
-request; the persistent-megakernel denoise (Stage 2) is not part of this release.
+`code/gr00t_p150/server/app.py` (FastAPI, `kind: tt-dit-server`); what it runs on the device is the port's **Stage-2**
+path: the vision tower, the LLM and the VL adapter as TTNN ops captured once into three Metal traces (`vision`, `llm`,
+`adapter`) and, in the fourth trace (`denoise`), the whole DiT flow-matching head — action encoder, 16 DiT blocks,
+decoder, 4 Euler steps — as **one persistent megakernel `generic_op`** (`TTPolicy.dit_backend="megakernel"`, bfp8_b
+weight arena in DRAM; the port's default since 2026-09-18). `GR00T_DIT_BACKEND=ttnn` serves the Stage-1 TTNN denoise
+(the 2026-09-14 release path) from the same image; both are reported by `GET /info -> stage`.
 
 | | |
 |---|---|
 | tt-metal tree the image is built from | `/home/deepgadget/experiments/gr00t/publish/tt-metal-668c2907575` — a clean local clone of tt-metal `main` @ `668c290757550588d0ce46b180c344a462a2aaf5` (`v0.79.0-dev20260914-1`), submodules populated, `git status` empty (the working clone `/home/deepgadget/experiments/gr00t/tt-metal` at the same commit has untracked dirs and would be recorded as dirty); torch pin 2.11.0 |
-| port code shipped | `code/models/experimental/gr00t/` = tt-metal ref `gr00t-p150-snapshot`, commit `5dc0517ee3bec090b7ead025001a343c2f63aa7b` (parent `668c290`; frozen 2026-09-14T23:36 KST), minus `__pycache__`, `.omc` tool state and 2,001 of the 2,004 per-test result JSONs (the three N1.5 e2e results the card cites and the `d0/d1/d2/mk_k1` summaries stay) |
+| port code shipped | `code/models/experimental/gr00t/` = tt-metal ref `gr00t-p150-snapshot-mk`, commit `009de892c895917ad6abe9a6c36518f00316ed4b` (parent `668c290`; frozen 2026-09-18T04:45 KST; megakernel tree `tt/megakernel/**` = the K4b-final checkpoint, sha256 listing `tests/tt/results/mk_tree_20260918-041439.sha256`, digest `7df61c35e9dc7857`), minus `__pycache__`, `.omc` tool state, the port's `generated/` runtime output and all but the 50 N1.5 result JSONs this repo cites (`tests/tt/results/`: the 13 `*_summary.md` + the K4b-checkpoint chain of 2026-09-18 04:08–04:37 KST + the 4 snapshot-validation JSONs; `benchmarks/results/`: 8 JSONs, §1) |
 | weights | `nvidia/GR00T-N1.5-3B` @ `869830fc749c35f34771aa5209f923ac57e4564e`: `model-0000{1,2,3}-of-00003.safetensors` (5,448,327,040 B of bf16 tensors; 5,448,460,016 B on disk in 3 files), `model.safetensors.index.json`, `config.json`, `experiment_cfg/metadata.json`, `LICENSE` — a pinned pointer, never in the image |
-| app | `gr00t_p150.server.app:app` (uvicorn `--lifespan on`; the lifespan does weights → tokenizer → device → model → trace capture → warm-up → READY) |
-| device recipe | `models.experimental.gr00t.tt.device.open_gr00t_device(trace_region_size=64 MiB, l1_small_size=32768, device_id=0)`; asserts the p150a's 11×10 compute grid and 8 DRAM banks; `TTPolicy(dtype_policy="mixed_dit", trace_layout="per_stage")` |
+| app | `gr00t_p150.server.app:app` (uvicorn `--lifespan on`; the lifespan does weights → tokenizer → policy → device → model → trace capture → warm-up → READY) |
+| device recipe | `models.experimental.gr00t.tt.model.open_model_device(policy, num_command_queues=2, trace_region_size=64 MiB, l1_small_size=32768, device_id=0)`: under the megakernel policy the worker L1 is opened 64 KiB smaller (`worker_l1_size` 1,461,248 − 65,536 = **1,395,712 B**; the kernel-config ring of the megakernel binaries, 136,192 B) and the VL adapter's intermediates are placed in DRAM; asserts the p150a's 11×10 compute grid and 8 DRAM banks; `TTPolicy(dtype_policy="mixed_dit", dit_backend="megakernel", mk_arena_dtype="bfp8_b", trace_layout="per_stage")` |
 | serving shape | embodiment `gr1` (slot 24), layout `gr1`: 1 camera (`ego_view`) × 1 frame × 256 image tokens, ≤ 102 text tokens, LLM sequence padded to 384, state 7/7/6/6 → 16 × [7, 7, 6, 6] actions, batch 1 |
 
 Below, `$PUB=/home/deepgadget/experiments/gr00t/publish` (this repo is `$PUB/GR00T-N1.5-3B-p150`),
@@ -25,15 +28,17 @@ Below, `$PUB=/home/deepgadget/experiments/gr00t/publish` (this repo is `$PUB/GR0
 ```
 tt-model.yaml                     authoring manifest (schema 5.1) -- the whole build/serve interface, incl. the model card text
 SERVING.md                        this file
-GPU_COMPARISON.md                 RTX 5090 vs p150a, matching definitions and caveats
+GPU_COMPARISON.md                 RTX 5090 vs p150a (megakernel and Stage-1 rows), matching definitions and caveats
 LICENSE                           the NVIDIA License (weights); LICENSE-NOTICE.md says what is under which licence
 media/demo_ego_view.png           the demo frame (256x256, GR1 sim robot_sim.PickNPlace traj 0 / step 100)
 media/demo_observation.json       the matching raw joint state + instruction (request body minus the image)
-media/demo_actions.png            served actions vs the fp32 reference for that request
+media/demo_actions.png            served actions vs the fp32 reference for that request (rendered from this image's smoke response)
 code/models/experimental/gr00t/   the tt-nn port (tt-metal tree layout): common/ (host pipeline: preprocessing, prompts,
-                                  normalisation, checkpoint, weight plan), tt/ (device model, layers, traces, megakernel WIP),
+                                  normalisation, checkpoint, weight plan), tt/ (device model, layers, traces; tt/megakernel/:
+                                  arena, core map, descriptors, dit_program + kernels/dit_kernel.cpp and ops/*.hpp),
                                   reference/ (fp32 torch reference), tests/, benchmarks/
-code/models/{common,tt_dit,demos}/   the 16 tt-metal files the port imports from the tree (source.code in tt-model.yaml)
+code/models/{common,tt_dit,demos}/   the 20 tt-metal files the port needs from the tree (source.code in tt-model.yaml): 16 Python
+                                  modules + the 4 deepseek_v3_b1/unified_kernels/*.hpp headers the megakernel's kernel source includes
 code/gr00t_p150/                  the policy server: server/{app,schemas,smoke_test}.py, demo/n15/ (canonical demo
                                   observation + golden actions + noise), assets/tokenizer/n15/ (vendored Qwen2 tokenizer)
 code/scripts/                     download_weights.sh, bench_http.py
@@ -44,8 +49,20 @@ tt_kernel_manifest.json, image/   written by `tt-model package`; uploaded by `tt
 The port keeps its tt-metal-tree absolute imports (`models.experimental.gr00t.*`, `models.tt_dit.*`,
 `models.common.*`): in the image the code lands at `/opt/tt-metal/<path>` with `PYTHONPATH=/opt/tt-metal`, and the
 image's tt-metal copy excludes `models/`, so everything under `models/` comes from this repo's `code/` (the port) and
-`source.code` (the 16 tree files). The server package `gr00t_p150` is shared with `changh95/GR00T-N1.6-3B-p150`; this
-repo ships only the N1.5 assets (`demo/n15`, `assets/tokenizer/n15`).
+`source.code` (the 20 tree files). The megakernel's kernel source is JIT-compiled at boot from
+`/opt/tt-metal/models/experimental/gr00t/tt/megakernel/kernels/dit_kernel.cpp`; it includes four headers of the
+tree's `models/demos/deepseek_v3_b1/unified_kernels/` (`kernel_op_api`, `kernel_utils`, `dataflow_utils`, `mcast`),
+which is why those are in `source.code` (their own includes live under `tt_metal/`, which the image ships). The server
+package `gr00t_p150` is shared with `changh95/GR00T-N1.6-3B-p150` and `changh95/GR00T-N1.7-3B-p150`; this repo ships
+only the N1.5 assets (`demo/n15`, `assets/tokenizer/n15`).
+
+Result JSONs shipped under `code/models/experimental/gr00t/benchmarks/results/` (the card's numbers ship with the code):
+`bench_e2e_n15_20260918-050550.json` (the snapshot's own megakernel benchmark, `commit 009de892…`, the card's port row),
+`bench_e2e_n15_20260918-043024.json` (the same benchmark on the live tree at the K4b checkpoint), `bench_e2e_n15_20260918-032243.json`
+(the same tree with `--dit-backend ttnn`: the Stage-1 reference the −16.8 % is measured against), `bench_mk_step_n15_20260918-035610.json`
+(the megakernel alone, 20 traced replays), `bench_load_n15_20260918-042919.json` (start-up cost with / without the ttnn DiT
+weights), and the 2026-09-14 Stage-1 documents `e2e_stage1_n15.json`, `denoise_stage1_n15.json`, `stage_ops_n15.json`
+(the previous release's card).
 
 ## 2. Run on the HOST for validation (no Docker)
 
@@ -63,8 +80,9 @@ export PYTHONPATH=$PWD/code:$TREE:$TREE/ttnn:$PUB/http-side      # code/ FIRST s
 export TT_METAL_CACHE=$HOME/.cache/tt-metal-cache-gr00t           # warm JIT kernels of the port's device sessions
 export HF_MODEL=nvidia/GR00T-N1.5-3B TT_WEIGHTS_REVISION=869830fc749c35f34771aa5209f923ac57e4564e
 export GR00T_VERSION=n15 TT_MESH_SHAPE=1x1 TT_DEVICE_ID=0 TT_METAL_VISIBLE_DEVICES=0
+export GR00T_DIT_BACKEND=megakernel GR00T_MK_ARENA_DTYPE=bfp8_b GR00T_NUM_CQS=2   # = the image's serve.env (= the port's defaults)
 export GR00T_TOKENIZER_DIR_N15=$PWD/code/gr00t_p150/assets/tokenizer/n15   # optional: the server picks the vendored dir itself
-export GR00T_TT_CACHE=$HOME/.cache/gr00t-tt                                # host .pt plan tier + .tensorbin device tier (~6.4 GB for N1.5)
+export GR00T_TT_CACHE=$HOME/.cache/gr00t-tt                                # host .pt plan tier + .tensorbin device tier (~2.5 GB under the megakernel for N1.5; 6.4 GB on the Stage-1 path)
 
 # import check, no device (what the image's verify.sh does):
 $TREE/python_env/bin/python -c "import gr00t_p150.server.app as a, sys; assert a.app and 'ttnn' not in sys.modules; print('ok')"
@@ -77,35 +95,40 @@ DEVICE_LOCK_TIMEOUT=14400 /home/deepgadget/experiments/gr00t/bin/with-device.sh
 ```
 
 Boot log landmarks (they drive `tt-model serve`'s checklist): `Loading weights: nvidia/GR00T-N1.5-3B @ 869830fc…` →
-`Tokenizer files: …` → `Opening device 0 (trace_region_size=67108864, l1_small_size=32768)` → `Loading pipeline:
-Gr00tTT.from_pretrained(n15, …)` → `Model built in N s (weights N s, 778 tensors, 3176 MB on device, cache path
-warm|cold)` → `Warming up: capture per_stage traces on the n15 demo observation, then N traced predict(s)` →
-`Warmup k/N: … ms (encode, device, decode)` → `Warm-up fidelity vs golden fp32: actions PCC … ` → `Warmup complete
-(capture s, first ms, steady ms; boot s) -- per_stage traces ['vision', 'llm', 'adapter', 'denoise']` → uvicorn
-`Application startup complete`. Any failure raises and uvicorn exits non-zero (no CPU fallback); the boot also **fails**
-if the warm-up actions' PCC vs the shipped golden is below `GR00T_WARMUP_PCC_MIN` (0.999). SIGTERM / Ctrl-C:
-`Releasing traces and device tensors` → `Closing device`.
-
-Measured host boots (warm `~/.cache/gr00t-tt` and warm `TT_METAL_CACHE`; `$PUB/logs/server_n15_*.log`): first boot of a
-process 14.6 s to READY (model built 9.3 s incl. 9.2 s reading the `.tensorbin` tier, warm + capture 2.5 s, 5 traced
-warm-ups 44.4–44.8 ms); an immediately following boot 4.5 s (0.5 s model build with the tiers in the page cache); on the
-frozen snapshot worktree 5.2 s (`server_n15_snapshot_20260914-235713.log`). Kernels are rebuilt once for a new
-`TT_METAL_HOME` path (~42 s for the 355 kernels of the four traces, `docs/publish/snapshot.md` §2.3).
-
-Expected smoke line (host, snapshot worktree, `$PUB/logs/smoke_n15_snapshot_20260914-235713.log`):
+`Tokenizer files: …` → `Opening device 0 (trace_region_size=67108864, l1_small_size=32768, num_command_queues=2,
+dit_backend=megakernel, mk_arena_dtype=bfp8_b)` → `Loading pipeline: Gr00tTT.from_pretrained(n15, …)` → `Model built in
+N s (weights N s, 499 tensors, 2652 MB on device, cache path warm|cold; dit_backend megakernel, arena bfp8_b, adapter_mem
+DRAM, head {mk_arena_plan_pack_s, mk_upload_s, …}; ttnn DiT weights skipped: 279 tensors / 524 MB)` → `Warming up: capture
+per_stage traces on the n15 demo observation, then N traced predict(s)` → `Warmup k/N: … ms (encode, device, decode)` →
+`Warm-up fidelity vs golden fp32: actions PCC …` → `Warmup complete (capture s, first ms, steady ms; boot s) -- per_stage
+traces ['vision', 'llm', 'adapter', 'denoise']` → uvicorn `Application startup complete`. Any failure raises and uvicorn
+exits non-zero (no CPU fallback); the boot also **fails** if the warm-up actions' PCC vs the shipped golden is below
+`GR00T_WARMUP_PCC_MIN` (0.999). SIGTERM / Ctrl-C: `Releasing traces and device tensors` → `Closing device`.
+
+Under the megakernel the ttnn DiT / precompute / encoder tensors are not uploaded (499 device tensors, 2,652 MB, instead
+of the Stage-1 path's 778 / 3,176 MB; `bench_load_n15_20260918-042919.json`); the bfp8_b weight arena the kernel streams
+from (16 blocks, 496 MB per step) is packed on the host and uploaded once at model build (`mk_arena_plan_pack_s` ≈ 0.9 s,
+`mk_upload_s` ≈ 1.0 s in the log line).
+
+Measured host boot of the megakernel path with a **fresh, empty `TT_METAL_CACHE`** (every kernel JIT-compiled, incl. the
+megakernel; warm `~/.cache/gr00t-tt` weight tiers; `GR00T_WARMUP_RUNS=5`; `$PUB/logs/server_n15_20260918-043232.log`,
+`tests/tt/results/mk_default_summary.md` §7): `Model built in 2.5 s` (weights 0.4 s), `warm_and_capture` 31.4 s, boot to
+READY **37.6 s**, 5 traced warm-ups 33.8 → 33.8 ms. Kernels are rebuilt once for a new `TT_METAL_HOME` path.
+
+Expected smoke line (host, megakernel default, same log stamp; `$PUB/logs/smoke_n15_20260918-043232.log`):
 
 ```
-PASS GR00T-N1.5-3B-p150 n15: PCC(actions)=0.999982 max|d|=0.0289 PCC(action_pred_valid)=0.999976 repeat_maxdiff=0.0e+00
-     seed_path_ok device_ms=42.95/42.16 total_ms=47.22/44.83 wall_ms=50
-     per_group[left_arm=0.99995/0.0214 right_arm=0.99997/0.0172 left_hand=0.94335/0.0179 right_hand=0.99999/0.0289]
+info: … stage=stage2-megakernel dit_backend=megakernel arena=bfp8_b cqs=2
+PASS GR00T-N1.5-3B-p150 n15: PCC(actions)=0.999980 max|d|=0.0296 PCC(action_pred_valid)=0.999976 repeat_maxdiff=0.0e+00
+     seed_path_ok device_ms=33.03/32.64 total_ms=37.02/35.0 wall_ms=40
+     per_group[left_arm=0.99994/0.0229 right_arm=0.99997/0.0172 left_hand=0.92834/0.0296 right_hand=0.99999/0.0289]
 ```
 
-The per-group PCC / max|d| equal the port's own device test (`code/models/experimental/gr00t/tests/tt/results/
-test_e2e_predict_actions_vs_golden_n15_20260914-230743.json`) to the printed precision; the traced policy is
-deterministic, so every run reproduces them bit-exactly. That JSON (and the two sibling `_2307*` JSONs the card cites)
-records the port's `two` trace layout, not the served `per_stage`; the same three tests re-run with `per_stage` on the
-frozen snapshot give identical values for every row (`docs/publish/snapshot.md` §4 of the port project), and the served
-`per_stage` actions equal the device test to the printed precision (§4).
+The per-group PCC / max|d| equal the port's own device test on the frozen snapshot
+(`code/models/experimental/gr00t/tests/tt/results/test_e2e_predict_actions_vs_golden_n15_20260918-044904.json`, `commit
+009de892…`) to the printed precision; the traced policy is deterministic, so every run reproduces them bit-exactly. The
+Stage-1 host smoke of the 2026-09-14 release (`smoke_n15_snapshot_20260914-235713.log`: PCC(actions) 0.999982, max|d|
+0.0289, device 42.95 / 42.16 ms) is the reference the megakernel figures are compared with in `GPU_COMPARISON.md`.
 
 Offline overrides: `GR00T_WEIGHTS_DIR=<dir with the snapshot files>` (shimmed into a private hub cache so the port's
 `snapshot_dir()` resolves to it), `GR00T_TOKENIZER_DIR=<dir>`.
@@ -121,7 +144,7 @@ uv venv --python 3.12 $PUB/depcheck-n15 -q
 uv pip install --python $PUB/depcheck-n15/bin/python --dry-run torch==2.11.0 fastapi uvicorn "pydantic>=2" pillow "numpy>=1.24.4,<2" \
    safetensors huggingface_hub "transformers==5.12.1" "opencv-python-headless==4.8.1.78" "torchvision==0.26.0" pytest \
    --extra-index-url https://download.pytorch.org/whl/cpu --index-strategy unsafe-best-match
-#   -> torch 2.11.0+cpu, torchvision 0.26.0+cpu, numpy 1.26.4, transformers 5.12.1, opencv-python-headless 4.8.1.78 (cp312 wheel), pytest 9.1.1 (logs/uv-dryrun-n15.log)
+#   -> torch 2.11.0+cpu, torchvision 0.26.0+cpu, numpy 1.26.4, transformers 5.12.1, opencv-python-headless 4.8.1.78 (cp312 wheel), pytest 9.1.1
 ```
 
 ## 3. Package, serve, push (Docker)
@@ -152,8 +175,8 @@ $ROOT/.venv/bin/tt-model push $PUB/build/gr00t-n1.5-3b-p150 --publish
 
 `serve` pre-downloads the pinned files (a cache hit on this host), then `docker run --user 0:0 --device
 /dev/tenstorrent --ipc host` with `/dev/hugepages-1G`, `~/.cache/huggingface` at `/hf` (rw, `HF_HOME=/hf`) and
-`~/.cache/tt-model/gr00t-n1.5-3b-p150/{cache,weights,tensors}` at `/cache` (`TT_METAL_CACHE`, JIT kernels),
-`/weight-cache` (`TT_DIT_CACHE_DIR`; the port's weight tiers go to `/weight-cache/gr00t-tt`, ~6.4 GB) and
+`~/.cache/tt-model/gr00t-n1.5-3b-p150/{cache,weights,tensors}` at `/cache` (`TT_METAL_CACHE`, JIT kernels incl. the
+megakernel), `/weight-cache` (`TT_DIT_CACHE_DIR`; the port's weight tiers go to `/weight-cache/gr00t-tt`, ~2.5 GB under the megakernel, 6.4 GB on the Stage-1 path) and
 `/tensor-cache`; it exports exactly `HF_MODEL=nvidia/GR00T-N1.5-3B`, `MESH_DEVICE=P150`, `TT_MESH_SHAPE=1x1` plus
 `serve.env` from `tt-model.yaml`, publishes the first free port from 20000 and waits ≤ 1800 s for `Application startup
 complete`. tt-cli users: `tt serve changh95/GR00T-N1.5-3B-p150` / `tt model stop changh95/GR00T-N1.5-3B-p150`
@@ -166,38 +189,41 @@ complete`. tt-cli users: `tt serve changh95/GR00T-N1.5-3B-p150` / `tt model stop
 
 ## 4. Container validation (the image built from this manifest)
 
-Image `tt-model/gr00t-n1.5-3b-p150:443501bd3b7a` (`sha256:443501bd3b7afce579bf743eeedd64809886653c29a9d7125614e8451cd61a7b`,
-3.2 GB as 46 OCI blobs), built 2026-09-14 15:17–15:30 UTC from the manifest above (`tt-model package`: tt-metal C++ build
-with a warm ccache ≈ 7 min, Python 3.12 venv, all 17 `verify:` lines + the launcher's own passed inside the image;
-13 min 17 s in total). Validated on the p150a on 2026-09-15 00:56–01:02 KST: three `tt-model serve` boots of this exact
-image, each followed by `smoke_test.py`, a 50-request `bench_http.py` and `tt-model stop`, all inside one device-lock hold
-per run (`publish/logs/validate-n15.run2.log` = boots 1–2, `validate-n15.run3.log` = boot 3; per-boot files
-`{serve,smoke,bench-http,container,info}-n15.{2,3}.*`; boot 1's per-file artefacts were overwritten by boot 3 and survive
-only inside `validate-n15.run2.log`).
+Image `tt-model/gr00t-n1.5-3b-p150:dfb9c3889ab1` (`sha256:dfb9c3889ab17cc1d8187085b27e854d0311fb974d729efe88db9bfdce51869f`, 3.2 GB (3,218,078,767 B in 46 content-addressed blobs)), built 2026-09-18 05:23–05:28 KST (`built.created_at` 2026-09-17T20:23:50+00:00) from the manifest above
+(`tt-model package`: tt-metal C++ build with a warm ccache, Python 3.12 venv, all 25 `verify:` lines + the launcher's own passed
+inside the image; 4 min 22 s with the two sibling GR00T builds sharing the host in total; `publish/logs/pkg-mk-n15.log`). Validated on the p150a on 2026-09-18 05:32–05:35 KST: two `tt-model serve`
+boots of this exact image inside **one** device-lock hold (`publish/scripts/validate_image_mk_n15.sh` via
+`run_validation_mk_n15_when_free.sh`, which enters the lock only when no `tt-model-*` container and no other process holds the
+card), each followed by `smoke_test.py`, a 50-request `bench_http.py` and `tt-model stop` by manifest path; per-boot files
+`publish/logs/{serve,container,info,smoke,bench-http}-mk-n15-c{1,2}-20260918-053217.*`, driver log `validate-mk-n15-20260918-053217.log`.
+Before boot 1 the package's `~/.cache/tt-model/gr00t-n1.5-3b-p150/{cache,weights}` of the Stage-1 release were moved aside, so
+boot 1 is the true cold boot of a fresh install (checkpoint → weight tiers, JIT of every kernel incl. the megakernel).
 
 | boot | caches (`~/.cache/tt-model/gr00t-n1.5-3b-p150/`) | host load | `tt-model serve` → READY | container boot (lifespan) | smoke | `bench_http.py` (10 warm-up + 50): device / server-side median (p90) | `tt-model stop` |
 |---|---|---|---|---|---|---|---|
-| 1 (**cold**) | `weights/` and `cache/` empty before; 6.4 GB + 411 MB after | two concurrent tt-metal image builds + CPU tests (load avg 16–110) | **1 min 43 s** (103.6 s wall) | 100.9 s: device open ≈ 4 s → `Model built in 8.9 s (… cache path cold)` (checkpoint → host plan → `.tensorbin`) → trace capture **87.3 s** incl. the JIT of the four traces' kernels (`riscv-tt-elf-g++` lines in the container log) → 3 warm-ups 50.9 / 45.8 / 101.5 ms | **PASS** — PCC(actions) 0.999982, max\|d\| 0.0289; PCC(`action_pred_valid`) 0.999976; device 46.96 / 42.21 ms, total 54.42 / 48.21 ms | 44.27 (55.35) / 53.59 (91.12) ms; encode median 3.2, p90 41 ms; client wall 58.1 (94.5) | clean SIGTERM shutdown 4.9 s |
-| 2 (warm, immediately after) | reused | same loaded host | 20.2 s (21.3 s wall) | 15.1 s: `Model built in 0.6 s (… cache path warm)`, capture 6.6 s, warm-ups 47.3 / 119.4 / 65.3 ms (encode spikes = host contention) | **PASS** — same fidelity; device 50.88 / 42.59 ms, total 67.44 / 45.45 ms | 44.51 (55.76) / 58.14 (96.86) ms; encode median 5.8, p90 38 ms; client wall 64.4 (104.2) | clean 3.1 s |
-| 3 (warm, quiet host) | reused | load avg ≈ 9, no other device or build work | **9.7 s** (10.0 s wall) | 7.6 s: model 0.6 s, capture 3.2 s, warm-ups 44.5 → 44.3 ms | **PASS** — same fidelity; device 42.99 / 42.19 ms, total 47.31 / 44.95 ms, client wall 50 ms | **42.43 (43.18) / 45.40 (49.20) ms**, min 41.66 / 44.46; decode 1.06, encode 1.63, decode_actions 0.14; client wall 47.6 (51.8) | clean 2.0 s |
-
-Fidelity is identical in every boot and identical to the host runs of §2 — the served `actions` JSON of the container is
-byte-for-byte the host run's (`smoke-n15.3.json` vs `smoke_n15_snapshot_20260914-235713.json`) — and equal to the
-port's device test `tests/tt/results/test_e2e_predict_actions_vs_golden_n15_20260914-230743.json` to the printed precision:
-`left_arm` 0.99995 / 0.0214, `right_arm` 0.99997 / 0.0172, `left_hand` 0.94335 / 0.0179 (abs-gated), `right_hand`
-0.99999 / 0.0289. The boot-3 benchmark is the card's "served" row; the device median 42.4 ms is 0.7 ms above the port's
-own `bench_e2e` device figure (41.68 ms: upload + traces + readback) and the loaded-host runs show that the extra
-latency and the p90 tails sit in the host-side stages (`encode`), not on the device.
-
-`GET /info` inside the container reports `weights.snapshot_dir = /hf/hub/models--nvidia--GR00T-N1.5-3B/snapshots/869830fc…`
-(the mounted HF cache), `source.tt_metal = {commit 668c290…, ttnn_dist_version 0.65.2.dev10011}`,
-`source.port_snapshot_commit = 5dc0517ee3b…`, `device_facts` grid 11×10 / DRAM 8×1 (`info-n15.3.json`).
-
-**Device-sharing note for this host.** The first validation attempt (00:32–00:40 KST) ran while another session's
-container was still holding the card after that session's lock hold had ended; `bin/with-device.sh`'s probe hung and
-reset the card twice, then refused to run. Since then every device entry goes through
-`publish/scripts/run_validation_n15_when_free.sh`, which waits until `docker ps` shows no `tt-model-*` container and the
-lock is free. Use the manifest path with `tt-model stop` / `logs` for an un-pushed package (`tt-model stop
+| 1 (**cold**) | `weights/` and `cache/` empty before; 343 MB `cache/` + 2.5 GB `weights/` after | load avg 7.8 at start, 31 at the end (two sibling image builds on the host) | **2 min 14 s (134.7 s wall: container start 31.6 s + lifespan)** | 97.7 s: `Model built in 29.9 s (weights 12.8 s, arena plan + pack 14.3 s; cache path cold)` → trace capture incl. the JIT of every kernel and the megakernel 55.7 s → 5 warm-ups 34.2 → 33.6 ms | **PASS** — PCC(actions) 0.999980, max\|d\| 0.0296; PCC(`action_pred_valid`) 0.999976; device 33.17 / 33.10 ms, total 36.90 / 35.56 ms | 32.59 (32.83) / 34.59 (35.05) ms; client wall 36.6 (37.2) | clean, 2.0 s |
+| 2 (warm) | reused | load avg 31 (same builds) | **11.2 s** | 9.0 s: `Model built in 2.6 s (… cache path warm)`, capture 2.9 s, warm-ups 33.6 → 33.4 ms | **PASS** — same fidelity; device 33.51 / 33.14 ms, total 37.21 / 35.52 ms | **32.52 (32.82) / 34.62 (35.39) ms**, min 32.29 / 34.09; decode 1.02, encode 0.90, decode_actions 0.10; client wall 36.3 (37.3) | clean, 2.0 s |
+
+Fidelity is identical in both boots and identical to the host run of §2 — the served `actions` JSON of the container is
+byte-for-byte the host run's (`smoke-mk-n15-c2-20260918-053217.json` vs `smoke_n15_20260918-043232.json`) — and equal to the
+port's device test on the frozen snapshot `tests/tt/results/test_e2e_predict_actions_vs_golden_n15_20260918-044904.json` to
+the printed precision: `left_arm` 0.99994 / 0.0229, `right_arm` 0.99997 / 0.0172, `left_hand` 0.92834 / 0.0296
+(abs-gated), `right_hand` 0.99999 / 0.0289. The boot-2 benchmark is the card's "served" row; its device median is
+0.1 ms above the port's own `bench_e2e` device figure on the snapshot (32.41 ms: upload + traces + readback,
+`bench_e2e_n15_20260918-050550.json`). The Stage-1 image of 2026-09-14 served 42.4 ms device / 45.4 ms server-side on
+the same host (`build-n15.md` §5); the megakernel image serves 9.9 ms less device time per request.
+
+`GET /info` inside the container reports `stage = {served: stage2-megakernel, dit_backend: megakernel, mk_arena_dtype:
+bfp8_b, num_command_queues: 2, cq1_uploads: true, worker_l1_size: 1395712, adapter_mem: DRAM, weights: {499 tensors,
+2652.2 MB, skipped_ttnn_dit: {279, 523.9 MB}}, head_timing_s}`, `weights.snapshot_dir = /hf/hub/models--nvidia--GR00T-N1.5-3B/snapshots/869830fc…`
+(the mounted HF cache), `source.tt_metal = {commit 668c290…}`, `source.port_snapshot_commit`, `device_facts` grid 11×10 /
+DRAM 8×1 (`info-mk-n15-c2-20260918-053217.json`).
+
+**Device-sharing note for this host.** The card is shared with other sessions; every device entry goes through
+`bin/with-device.sh` (flock + a ttnn probe) and, for containers, through `publish/scripts/run_validation_mk_n15_when_free.sh`,
+which waits until `docker ps` shows no `tt-model-*` container, `fuser /dev/tenstorrent/0` is empty and the lock is free
+(a container left running after its lock hold ends would otherwise be reset by the next caller's probe — `build-n15.md`
+§5.0). Use the manifest path with `tt-model stop` / `logs` for an un-pushed package (`tt-model stop
 changh95/GR00T-N1.5-3B-p150` only works after `pull`).
 
 ## 5. Request / response contract
@@ -205,7 +231,7 @@ changh95/GR00T-N1.5-3B-p150` only works after `pull`).
 | route | returns |
 |---|---|
 | `GET /health` | `{"status": "ok" \| "starting", "model": "GR00T-N1.5-3B-p150", "device": "blackhole:0"}` — 200 always; `ok` only after the warm-up |
-| `GET /info` | model / version / task / hardware / `stage` (`stage1-ttnn-traces` + megakernel note) / `weights` (repo, revision, snapshot dir actually loaded) / `source` (port snapshot commit, server version, tt-metal commit, ttnn dist version) / `inputs` (the full observation contract) / `outputs` / `noise.default_seed` / `limits` / `device_facts` (11×10 grid, 8 DRAM banks) / `warmup_latency_ms` / `warmup_fidelity_vs_golden` / `license` |
+| `GET /info` | model / version / task / hardware / `stage` (`served`: `stage2-megakernel` \| `stage1-ttnn-traces`, `dit_backend`, `mk_arena_dtype`, `num_command_queues`, `cq1_uploads`, `worker_l1_size`, `adapter_mem`, `weights` incl. the skipped ttnn DiT set, `head_timing_s`, a note) / `policy` (the resolved `TTPolicy`) / `weights` (repo, revision, snapshot dir actually loaded) / `source` (port snapshot commit, server version, tt-metal commit, ttnn dist version) / `inputs` (the full observation contract) / `outputs` / `noise.default_seed` / `limits` / `device_facts` (11×10 grid, 8 DRAM banks) / `warmup_latency_ms` / `warmup_fidelity_vs_golden` / `license` |
 | `GET /v1/models` | OpenAI-shaped stub `{"object": "list", "data": [{"id": "nvidia/GR00T-N1.5-3B", "object": "model", "owned_by": "changh95"}]}` so the tt-model ready card / `tt-model curl` do not 404; this is not a chat API |
 | `GET /demo` | `{"request": <ready-to-post demo body>, "expected": <fp32 reference actions + gates>}` |
 | `POST /predict` | one observation → one 16-step action chunk (below) |
@@ -231,45 +257,67 @@ Response (200):
  "normalized": false, "embodiment": "gr1", "embodiment_id": 24, "version": "n15", "model": "GR00T-N1.5-3B-p150", "layout": "gr1",
  "images": {"ego_view": {"frames": 1, "received_hw": [256, 256]}}, "seq_len": 296, "prompt_tokens": 14, "state_dtype": "float64",
  "noise_source": "seed" | "client", "seed": 0 | null,
- "timing_ms": {"decode": 1.5, "encode": 2.6, "device": 43.0, "decode_actions": 0.1, "total": 47.2}}
+ "timing_ms": {"decode": 1.47, "encode": 2.02, "device": 33.51, "decode_actions": 0.15, "total": 37.21}}
 ```
 
 `actions` are physical joint targets (radians) — `(y + 1) / 2 · (max − min) + min` with the checkpoint's
 `experiment_cfg/metadata.json` GR1 statistics — exactly what `Gr00tPolicy.get_action` returns. `timing_ms`: `decode` =
 base64 + image decode + validation; `encode` = the port's host preprocessing; `device` = `predict_normalized` (static-shape
-asserts, input writes, 4 × `execute_trace`, one blocking readback); `decode_actions` = un-normalisation; `total` = the
-whole handler. Errors: **400** structural problems (`{"detail": {"errors": [...], "inputs": {...}}}`), undecodable image,
-bad state / instruction / noise, other embodiment; **503** while starting; **500** `Type: text` on a device failure.
-Handlers are synchronous and serialised on one lock; batch 1.
+asserts, input writes on CQ 1, 4 × `execute_trace` — the fourth is the megakernel —, one blocking readback);
+`decode_actions` = un-normalisation; `total` = the whole handler. Errors: **400** structural problems (`{"detail":
+{"errors": [...], "inputs": {...}}}`), undecodable image, bad state / instruction / noise, other embodiment; **503** while
+starting; **500** `Type: text` on a device failure. Handlers are synchronous and serialised on one lock; batch 1.
 
 ## 6. Environment (read in the lifespan, never at import)
 
 | variable | container value (`serve.env`) | meaning |
 |---|---|---|
 | `HF_MODEL` / `TT_WEIGHTS_REVISION` | `nvidia/GR00T-N1.5-3B` (launcher) / `869830fc…` | must be the version's pinned repo and sha (both are asserted against the port's own `configs.HF_REPOS` / `HF_SNAPSHOT_SHAS`) |
-| `GR00T_VERSION` | `n15` | the server code is shared by both GR00T packages and never guesses |
+| `GR00T_VERSION` | `n15` | the server code is shared by the three GR00T packages and never guesses |
 | `GR00T_EMBODIMENT` / `GR00T_LAYOUT` | `gr1` / `gr1` | the only device-validated layout of this release |
 | `GR00T_POLICY` / `GR00T_TRACE_LAYOUT` | `mixed_dit` / `per_stage` | `TTPolicy` knobs; `bf16` / `mixed` and `two` exist in the port but are not the benchmarked defaults |
-| `GR00T_TRACE_REGION_SIZE` / `GR00T_L1_SMALL_SIZE` | default 67108864 / 32768 | the port's validated `open_gr00t_device` values |
+| `GR00T_DIT_BACKEND` | `megakernel` (= the port's default, pinned) | `TTPolicy.dit_backend`: `megakernel` = the denoise trace is one persistent `generic_op` (Stage 2); `ttnn` = the Stage-1 sequence of TTNN ops (for A/B; the device is then opened with the firmware `worker_l1_size` and the adapter's intermediates in L1). Reported in `/info -> stage` |
+| `GR00T_MK_ARENA_DTYPE` | `bfp8_b` (= what `auto` resolves to for n15) | dtype of the megakernel's DRAM weight arena: `auto` / `bf16` / `bfp8_b`; the bf16 arena is 2.66 ms per step instead of 2.22 (`tests/tt/results/mk_k4b_summary.md` §0) |
+| `GR00T_NUM_CQS` | `2` | command queues of `open_model_device`: `2` = per-call input writes on CQ 1 overlap the trace replays (the benchmarked path; bit-equal to `1` per `tests/tt/results/test_mk_2cq_bit_equality_n15_20260918-041502.json`) |
+| `GR00T_TRACE_REGION_SIZE` / `GR00T_L1_SMALL_SIZE` | default 67108864 / 32768 | the port's validated `open_model_device` values |
+| `GR00T_MK_L1_CUT_KIB` (port) | unset (= 64) | the worker-L1 cut the megakernel's kernel-config ring needs; read by the port's `open_model_device`, not by the server. Changing it invalidates the validated configuration |
 | `GR00T_TOKENIZER_DIR_N15` | `/opt/tt-metal/gr00t_p150/assets/tokenizer/n15` | vendored Qwen2 tokenizer (not in the weights repo); `GR00T_TOKENIZER_DIR` overrides |
 | `GR00T_TT_CACHE` | `/weight-cache/gr00t-tt` | host `.pt` plan tier + `.tensorbin` device tier (persisted under `~/.cache/tt-model/gr00t-n1.5-3b-p150/weights`) |
 | `GR00T_PROJECT_ROOT` / `GR00T_GOLDEN_ROOT` | `/nonexistent` | the port's dev-box defaults; nothing on the serve path needs them |
-| `GR00T_WARMUP_RUNS` / `GR00T_WARMUP_PCC_MIN` | `3` / default 0.999 | traced predicts before READY; boot fails below the PCC floor vs the shipped golden |
+| `GR00T_WARMUP_RUNS` / `GR00T_WARMUP_PCC_MIN` | `5` / `0.999` | traced predicts before READY; boot fails below the PCC floor vs the shipped golden |
 | `GR00T_DEFAULT_SEED`, `GR00T_TORCH_THREADS`, `GR00T_WEIGHTS_DIR` | unset | request default seed (0), torch intra-op threads, offline snapshot dir |
 | `TT_MESH_SHAPE`, `TT_DEVICE_ID`, `TT_METAL_VISIBLE_DEVICES` | `1x1` (launcher), `0`, `0` | one chip; any other mesh shape is refused |
 
 ## 7. Caveats
 
-- **Stage-1 path only.** TTNN ops in Metal traces; the persistent-megakernel denoise (K1 streaming kernel: 464 GB/s bf16 /
-  414 GB/s bfp8, `code/models/experimental/gr00t/tests/tt/results/mk_k1_summary.md`) is not integrated into `Gr00tTT` and not served.
+- **Stage-2 path.** The denoise is one persistent megakernel `generic_op` (K4b-final kernel: bfp8 weights streamed from a
+  DRAM arena once per step, 110 Tensix cores, dual-NoC hub multicast, distributed LayerNorm, per-head flash SDPA split over
+  three cores for the long-key cross blocks; `tests/tt/results/mk_k4b_summary.md`, `mk_k5_summary.md`, `mk_default_summary.md`).
+  Consequences: (1) the device is opened with a 64 KiB smaller worker L1 (`worker_l1_size` 1,395,712 B) so the kernel-config
+  ring holds the ≈ 131.7 KB compute-core program (4.5 KB of headroom under the 136,192 B ring: any kernel growth must be
+  re-checked), and the VL adapter's intermediates are placed in DRAM (+0.38 ms on the `adapter` trace: 4.86 vs 4.48 ms);
+  (2) the DiT residual-stream intermediates (`dit_block{i}_out`, `dit_out[k]`) are not observable from the served model —
+  the untraced tap set is the backbone / adapter taps + `action_decoder_out[k]`, `pred_velocity[k]` and the action rows
+  (30 rows instead of the Stage-1 path's 58); (3) the e2e gain over the Stage-1 denoise on the same tree is −16.8 %
+  (40.38 → 33.59 ms; denoise trace 16.02 → 8.93 ms, −44 %), not the plan's −30 %.
+- **Fidelity vs the Stage-1 path.** Gate outcomes are equal on 8 of the 9 N1.5 observations (the demo and 7 of the 8 multi-sample
+  goldens); the megakernel's only extra miss is `unnormalized_actions.right_arm` on `t3_s300_seed2` at PCC 0.999869 vs
+  the 0.9999 gate (max|d| 0.029 rad within its 0.038 tolerance; recorded as a known marginal row with floor 0.9998 in
+  `tests/tt/test_mk_e2e.py`; `tests/tt/results/test_mk_e2e_backends_agree_t3_s300_seed2_n15_20260918-042413.json`). On the
+  demo observation megakernel vs TTNN `action_pred_normalized` PCC is 0.999997 (`…_canonical_n15_20260918-042408.json`).
 - **Shape-locked traces.** One embodiment (`gr1`), one camera, one frame, ≤ 102 text tokens (LLM S padded to 384), batch 1.
   Other embodiments / cameras / longer instructions → 400 before anything reaches the device. Concurrent requests queue on one lock.
 - **Deterministic noise.** Actions are a deterministic function of (observation, seed | noise); the reference noise of the
-  demo is not seed-reproducible, so the smoke test posts it as `noise`. Same observation + same noise → bit-identical actions.
+  demo is not seed-reproducible, so the smoke test posts it as `noise`. Same observation + same noise → bit-identical actions;
+  traced == untraced and 2 CQs == 1 CQ bit for bit.
 - **Base checkpoint.** Outputs are the GR1 sim data-config actions of the released N1.5 checkpoint; a real robot needs NVIDIA's post-training.
 - **First container boot is cold**: the port converts the 5.4 GB checkpoint into its `.pt` plan tier and the `.tensorbin`
-  device tier under `/weight-cache/gr00t-tt` and JITs the kernels of the four traces into `/cache`; both persist under
-  `~/.cache/tt-model/gr00t-n1.5-3b-p150/`, so later boots are seconds (measured values in §4).
+  device tier under `/weight-cache/gr00t-tt` and JITs every kernel of the four traces — incl. the megakernel — into `/cache`;
+  both persist under `~/.cache/tt-model/gr00t-n1.5-3b-p150/`, so later boots are seconds (measured values in §4). A cache
+  directory left by the Stage-1 image is reused (its kernels stay, the megakernel is added).
+- **One megakernel model per process.** Building a second megakernel `Gr00tTT` after releasing the first on the same open
+  device hung the card on N1.6 during development (`mk_default_summary.md` §9); the server builds exactly one model per
+  process and never rebuilds it.
 - **Python 3.12 image vs 3.10 tree venv.** Host runs are an approximation; the image is authoritative (its `verify.sh`
   imports the app and the port with no device; the served smoke in §4 is the proof).
 - **Weights licence.** `nvidia/GR00T-N1.5-3B` is under the NVIDIA License (non-commercial: research or evaluation use);

tt-model.yaml

@@ -1,5 +1,8 @@
 # SPDX-License-Identifier: Apache-2.0
-# tt-model-manager container manifest (schema 5.1) for NVIDIA Isaac GR00T N1.5-3B (Stage-1 TTNN path) on Blackhole.
+# tt-model-manager container manifest (schema 5.1) for NVIDIA Isaac GR00T N1.5-3B on Blackhole -- Stage-2 release:
+# the DiT denoise runs as ONE persistent megakernel generic_op (TTPolicy.dit_backend="megakernel", bfp8_b weight
+# arena, the port's default since 2026-09-18); vision / llm / adapter stay TTNN ops in Metal traces.  The Stage-1
+# TTNN denoise is selectable with GR00T_DIT_BACKEND=ttnn.
 #
 # Run EVERY tt-model command from this directory: `source.tt_metal` and `extra_code[].root: code`
 # resolve against the process CWD, not against this file.  Rootless Docker on this box needs
@@ -38,7 +41,7 @@ source:
   # three submodules populated and `git status --porcelain` empty, so the image is built from exactly the commit the
   # card links to (the working clone has untracked dirs and would be recorded as a dirty tree).
   tt_metal: /home/deepgadget/experiments/gr00t/publish/tt-metal-668c2907575
-  # tt-metal-relative files the port imports from the tree OTHER than itself, with their import closure
+  # tt-metal-relative files the port needs from the tree OTHER than itself, with their import closure
   # (the image's tt-metal copy excludes models/, so everything under models/ must be listed here or in extra_code).
   code:
     - models/common/utility_functions.py                         # models.tt_dit.utils.matmul imports is_blackhole (needs pytest!)
@@ -56,13 +59,22 @@ source:
     - models/tt_dit/parallel/config.py
     - models/tt_dit/parallel/manager.py
     - models/tt_dit/encoders/qwen3vl/vision_qwen3vl.py           # tt/qwen3vl_vision.py (lazy; not on the N1.5 serve path)
-    - models/demos/deepseek_v3_b1/unified_kernel_descriptor.py   # tt/megakernel/descriptors.py (lazy; Stage-2 work, not served)
+    - models/demos/deepseek_v3_b1/unified_kernel_descriptor.py   # tt/megakernel/descriptors.py: the ProgramDescriptor builder of the megakernel (served)
+    # Device-side headers the megakernel's kernel source includes (JIT-compiled at boot from
+    # /opt/tt-metal/models/experimental/gr00t/tt/megakernel/kernels/dit_kernel.cpp -> ../../../../../demos/deepseek_v3_b1/unified_kernels/*.hpp
+    # and, from ops/allgather.hpp, "models/demos/deepseek_v3_b1/unified_kernels/mcast.hpp" through -I$TT_METAL_HOME).
+    # Their own includes (tt_metal/tools/profiler/kernel_profiler.hpp, api/compute/blank.h, api/dataflow/dataflow_api.h,
+    # api/compute/experimental/deepseek_compute_kernel_hw_startup.h) live under tt_metal/, which the image ships.
+    - models/demos/deepseek_v3_b1/unified_kernels/kernel_op_api.hpp
+    - models/demos/deepseek_v3_b1/unified_kernels/kernel_utils.hpp
+    - models/demos/deepseek_v3_b1/unified_kernels/dataflow_utils.hpp
+    - models/demos/deepseek_v3_b1/unified_kernels/mcast.hpp
   # This repo's own code (lands at /opt/tt-metal/<path>; PYTHONPATH=/opt/tt-metal).
   extra_code:
     - root: code
       paths:
-        - models          # code/models/experimental/gr00t: the frozen snapshot of the port (tt-metal ref gr00t-p150-snapshot, 5dc0517ee3b)
-        - gr00t_p150      # the policy server, its smoke test, the demo observation and the vendored Qwen2 tokenizer files
+        - models          # code/models/experimental/gr00t: the frozen megakernel-release snapshot of the port (tt-metal ref gr00t-p150-snapshot-mk, 009de892c895; K4b-final kernel tree 7df61c35e9dc7857)
+        - gr00t_p150      # the policy server (megakernel default, open_model_device), its smoke test, the demo observation and the vendored Qwen2 tokenizer files
         - scripts         # download_weights.sh, bench_http.py
   ubuntu: "22.04"
   python: "3.12"
@@ -95,35 +107,47 @@ serve:
     GR00T_LAYOUT: "gr1"                  # static layout: 1 image x 256 tokens, L_max 102 text tokens, LLM sequence padded to 384
     GR00T_POLICY: "mixed_dit"            # TTPolicy.dtype_policy: DiT block matmul weights bfp8_b, everything else bf16 (validated default)
     GR00T_TRACE_LAYOUT: "per_stage"      # four Metal traces vision / llm / adapter / denoise (the benchmarked layout)
+    GR00T_DIT_BACKEND: "megakernel"      # TTPolicy.dit_backend: the denoise trace = one persistent megakernel generic_op (the port's default; pinned). "ttnn" = the Stage-1 TTNN op sequence
+    GR00T_MK_ARENA_DTYPE: "bfp8_b"       # the megakernel's DRAM weight arena (what "auto" resolves to for n15: MK_ARENA_DTYPE_DEFAULT); "bf16" selectable
+    GR00T_NUM_CQS: "2"                   # command queues of tt.model.open_model_device: per-call input writes on CQ 1 overlap the traces (bit-equal to 1 CQ, tests/tt/test_mk_2cq.py)
     GR00T_TOKENIZER_DIR_N15: "/opt/tt-metal/gr00t_p150/assets/tokenizer/n15"
     GR00T_PROJECT_ROOT: "/nonexistent"   # the port's dev-box default for project-relative paths; nothing on the serve path needs it
     GR00T_GOLDEN_ROOT: "/nonexistent"    # tests only (defensive)
     GR00T_TT_CACHE: "/weight-cache/gr00t-tt"   # host plan (.pt) + device-layout (.tensorbin) weight tiers -> ~/.cache/tt-model/gr00t-n1.5-3b-p150/weights
-    GR00T_WARMUP_RUNS: "3"               # traced predict() calls after trace capture, before READY
+    GR00T_WARMUP_RUNS: "5"               # traced predict() calls after trace capture, before READY (the megakernel-default smoke configuration)
+    GR00T_WARMUP_PCC_MIN: "0.999"        # boot FAILS when the warm-up actions' PCC vs the shipped fp32 golden is below this
 
 # Build-time assertions, run INSIDE the finished image as uid 1000, no device, no weights, no HF_MODEL.
 verify:
   - "import gr00t_p150.server.app as a; assert a.app; import sys; assert 'ttnn' not in sys.modules, 'server import must be side-effect free'"
   - "import models.experimental.gr00t.tt.model as m; assert m.Gr00tTT"
   - "from models.experimental.gr00t.common import preprocessing, normalization, prompts, checkpoint, weights, configs; assert configs.HF_SNAPSHOT_SHAS['n15'] == '869830fc749c35f34771aa5209f923ac57e4564e'"
+  - "from models.experimental.gr00t.tt.policy import TTPolicy, DEFAULT_DIT_BACKEND, MK_ARENA_DTYPE_DEFAULT; p = TTPolicy(); assert p.dit_backend == 'megakernel' == DEFAULT_DIT_BACKEND and p.mk_arena_dtype == 'auto' and MK_ARENA_DTYPE_DEFAULT['n15'] == 'bfp8_b', (p.dit_backend, p.mk_arena_dtype)"
+  - "import os; os.environ['GR00T_DIT_BACKEND'] = 'ttnn'; from models.experimental.gr00t.tt.policy import TTPolicy; assert TTPolicy().with_env_overrides().dit_backend == 'ttnn' and TTPolicy().dit_backend == 'megakernel'"
+  - "from models.experimental.gr00t.tt.model import open_model_device, device_worker_l1_size, MK_L1_CUT_BYTES_DEFAULT, ADAPTER_MEM_BY_BACKEND; assert MK_L1_CUT_BYTES_DEFAULT == 65536 and ADAPTER_MEM_BY_BACKEND['megakernel'] == 'DRAM'"
+  - "import models.experimental.gr00t.tt.megakernel.dit_program, models.experimental.gr00t.tt.megakernel.arena, models.experimental.gr00t.tt.megakernel.core_map; from models.experimental.gr00t.tt.megakernel.descriptors import KERNEL_SOURCE, L1_USABLE_BYTES; from pathlib import Path; assert Path('/opt/tt-metal', KERNEL_SOURCE).is_file(), KERNEL_SOURCE"
+  - "from pathlib import Path; k = Path('/opt/tt-metal/models/experimental/gr00t/tt/megakernel/kernels'); assert all((k / 'ops' / f).is_file() for f in ('allgather.hpp', 'block_matmul.hpp', 'dist_layernorm.hpp', 'euler_head.hpp', 'head_sdpa.hpp', 'row_layernorm.hpp', 'weight_stream.hpp')), sorted(p.name for p in (k / 'ops').iterdir())"
+  - "from pathlib import Path; d = Path('/opt/tt-metal/models/demos/deepseek_v3_b1/unified_kernels'); assert all((d / f).is_file() for f in ('kernel_op_api.hpp', 'kernel_utils.hpp', 'dataflow_utils.hpp', 'mcast.hpp')), sorted(p.name for p in d.iterdir()) if d.is_dir() else 'missing dir'"
+  - "from pathlib import Path; assert Path('/opt/tt-metal/tt_metal/tools/profiler/kernel_profiler.hpp').is_file() and Path('/opt/tt-metal/tt_metal/hw/inc/api/compute/experimental/deepseek_compute_kernel_hw_startup.h').is_file() and Path('/opt/tt-metal/tt_metal/hw/inc/api/compute/blank.h').is_file()"
   - "import pytest, models.common.utility_functions; import models.tt_dit.utils.matmul as mm; assert mm.get_matmul_config"
   - "import models.demos.deepseek_v3_b1.unified_kernel_descriptor as u; assert u.UnifiedKernelDescriptor"
   - "import models.tt_dit.encoders.qwen3vl.vision_qwen3vl"
-  - "import models.experimental.gr00t.tt.layers, models.experimental.gr00t.tt.qwen3vl_vision, models.experimental.gr00t.tt.megakernel.descriptors"
+  - "import models.experimental.gr00t.tt.layers, models.experimental.gr00t.tt.qwen3vl_vision, models.experimental.gr00t.tt.action_head"
+  - "from gr00t_p150.server.app import STAGE_SERVED, DEFAULT_NUM_CQS, DIT_BACKENDS; assert STAGE_SERVED['megakernel'] == 'stage2-megakernel' and STAGE_SERVED['ttnn'] == 'stage1-ttnn-traces' and DEFAULT_NUM_CQS == 2 and set(DIT_BACKENDS) == {'ttnn', 'megakernel'}"
   - "import safetensors, huggingface_hub, cv2, numpy, PIL; assert int(numpy.__version__.split('.')[0]) < 2, numpy.__version__"
   - "import transformers; assert transformers.__version__ == '5.12.1', transformers.__version__"
   - "import torchvision; assert torchvision.__version__.startswith('0.26.0'), torchvision.__version__"
   - "from pathlib import Path; d = Path('/opt/tt-metal/gr00t_p150/assets/tokenizer/n15'); assert all((d / f).is_file() for f in ('vocab.json', 'merges.txt', 'tokenizer_config.json', 'special_tokens_map.json', 'added_tokens.json')), sorted(p.name for p in d.iterdir())"
   - "import os; os.environ['GR00T_TOKENIZER_DIR_N15'] = '/opt/tt-metal/gr00t_p150/assets/tokenizer/n15'; from models.experimental.gr00t.common.prompts import get_tokenizer; t = get_tokenizer('n15'); assert t.convert_tokens_to_ids('<IMG_CONTEXT>') == 151669"
-  - "from pathlib import Path; assert Path('/opt/tt-metal/models/experimental/gr00t/common/golden_tap_map.json').is_file()"
-  - "from pathlib import Path; assert Path('/opt/tt-metal/gr00t_p150/server/smoke_test.py').is_file() and Path('/opt/tt-metal/scripts/bench_http.py').is_file()"
+  - "from pathlib import Path; assert Path('/opt/tt-metal/models/experimental/gr00t/common/golden_tap_map.json').is_file() and Path('/opt/tt-metal/models/experimental/gr00t/tests/tt/gates/gates_multi.json').is_file()"
+  - "from pathlib import Path; assert Path('/opt/tt-metal/gr00t_p150/server/smoke_test.py').is_file() and Path('/opt/tt-metal/scripts/bench_http.py').is_file() and Path('/opt/tt-metal/scripts/download_weights.sh').is_file()"
   - "from pathlib import Path; d = Path('/opt/tt-metal/gr00t_p150/demo/n15'); assert all((d / f).is_file() for f in ('observation.json', 'noise.json', 'expected.json', 'frames/ego_view_t0.png')), sorted(str(p) for p in d.rglob('*'))"
-  - "import ttnn; assert all(hasattr(ttnn, a) for a in ('begin_trace_capture', 'end_trace_capture', 'execute_trace', 'release_trace', 'copy_host_to_device_tensor', 'MinimalMatmulConfig'))"
+  - "import ttnn; assert all(hasattr(ttnn, a) for a in ('begin_trace_capture', 'end_trace_capture', 'execute_trace', 'release_trace', 'copy_host_to_device_tensor', 'MinimalMatmulConfig', 'generic_op', 'ProgramDescriptor', 'ComputeConfigDescriptor', 'CBDescriptor', 'SemaphoreDescriptor', 'UnpackToDestMode'))"
   - "from gr00t_p150.server.app import parse_mesh_shape as p; assert p('1x1') == p('(1, 1)') == p('1,1') == (1, 1)"
 
 card:
   description: |
-    NVIDIA Isaac GR00T N1.5 (3B vision-language-action policy) running on one Tenstorrent Blackhole p150a via tt-nn: one camera frame, proprioceptive state and a task instruction in, a 16-step chunk of GR1 arm and hand joint targets out. Stage-1 release: TTNN ops replayed from Metal traces (DiT weights bfp8, everything else bf16); the persistent-megakernel denoise is in progress and not in this image.
+    NVIDIA Isaac GR00T N1.5 (3B vision-language-action policy) running on one Tenstorrent Blackhole p150a via tt-nn: one camera frame, proprioceptive state and a task instruction in, a 16-step chunk of GR1 arm and hand joint targets out. Stage-2 release: vision tower, LLM and VL adapter as TTNN ops replayed from Metal traces (bf16); the whole DiT denoise (4 steps × 16 blocks) as **one persistent megakernel** on 110 Tensix cores, streaming its bfp8 weights from a DRAM arena once per step. `GR00T_DIT_BACKEND=ttnn` selects the Stage-1 TTNN denoise.
     Weights: [nvidia/GR00T-N1.5-3B](https://huggingface.co/nvidia/GR00T-N1.5-3B) · Paper: [arXiv:2503.14734](https://arxiv.org/abs/2503.14734) · Upstream code: [NVIDIA/Isaac-GR00T](https://github.com/NVIDIA/Isaac-GR00T) (`n1.5-release`) · Port: `code/models/experimental/gr00t` and the server `code/gr00t_p150` in this repo
   quickstart: |
     ### Run with tt-cli
@@ -147,14 +171,15 @@ card:
     ### Response
 
     ```json
-    {"actions": {"left_arm":   [[0.0069, 0.2107, 0.0483, -1.8136, -0.1264, -0.0548, 0.0147], "... 16 steps"],
-                 "right_arm":  [[-0.0177, -0.1731, 0.3702, -1.7429, 0.4968, -0.0567, 0.9849], "..."],
-                 "left_hand":  [[0.0081, 0.0155, 0.0033, 0.0064, 0.0169, 0.0438], "..."],
-                 "right_hand": [[-1.4880, -1.4915, -1.4781, -1.4810, -2.9609, 2.9904], "..."]},
-     "action_horizon": 16, "embodiment": "gr1", "seq_len": 296,
-     "timing_ms": {"decode": 1.5, "encode": 2.6, "device": 43.0, "decode_actions": 0.1, "total": 47.2}}
+    {"actions": {"left_arm":   [[0.0032, 0.2093, 0.0567, -1.8143, -0.1239, -0.0563, 0.0177], "... 16 steps"],
+                 "right_arm":  [[-0.0157, -0.1731, 0.3713, -1.7457, 0.4968, -0.0574, 0.9908], "..."],
+                 "left_hand":  [[0.0081, 0.0121, 0.0042, 0.0064, 0.0169, 0.0380], "..."],
+                 "right_hand": [[-1.4850, -1.4915, -1.4734, -1.4743, -2.9550, 2.9947], "..."]},
+     "action_horizon": 16, "embodiment": "gr1", "seq_len": 296, "noise_source": "client", "seed": null,
+     "timing_ms": {"decode": 1.47, "encode": 2.02, "device": 33.51, "decode_actions": 0.15, "total": 37.21}}
     ```
 
+    - The example is the demo request with the reference noise, as `GET /demo` returns it (`noise_source: "client"`, `seed: null`): the smoke response of this image's second (warm) boot, `timing_ms` included.
     - `actions` are physical, un-normalised joint targets (radians), 16 steps × [7, 7, 6, 6] dims, as `Gr00tPolicy.get_action` returns them; `timing_ms.device` covers upload, the trace replays and the readback.
 
     ### Demo
@@ -167,20 +192,23 @@ card:
 
     | Metric | Value |
     |---|---:|
-    | Action chunk vs the fp32 reference (same observation, same initial noise) | PCC `left_arm` 0.99995 · `right_arm` 0.99997 · `right_hand` 0.99999 · `left_hand` max\|d\| 0.018 rad (near-constant group) · normalised chunk 0.99998 |
-    | Intermediate taps vs the fp32 reference (57 gated taps, from pixels) | 58/58 rows pass |
-    | Traced vs untraced, determinism | bit-identical; 5 trace replays bit-identical |
-    | Inference on p150a (warm, batch 1, median of 50) | **43.4 ms end-to-end** = host encode 1.5 + upload 3.8 + traces 37.8 (vision 8.7 · LLM 8.7 · adapter 4.5 · denoise 16.0) + readback and decode 0.3 |
-    | Served over HTTP by this image (warm, batch 1) | 42.4 ms device · 45.4 ms server-side incl. PNG decode |
-    | Same policy on an RTX 5090 (same host, NVIDIA's unmodified `Gr00tPolicy`, eager PyTorch bf16 + flash-attn 2) | 58.0 ms end-to-end (model 55.4) → p150a 1.34× faster |
+    | Action chunk vs the fp32 reference (same observation, same initial noise) | PCC `left_arm` 0.99994 · `right_arm` 0.99997 · `right_hand` 0.99999 · `left_hand` max\|d\| 0.030 rad (near-constant group) · normalised chunk 0.99998 |
+    | Intermediate taps vs the fp32 reference (the 28 taps the megakernel path exposes, from pixels; DiT intermediates not observable inside the megakernel) | 30/30 rows pass (29 gated); tightest margin `vit_block_first` 0.99994 vs 0.9999 |
+    | Megakernel vs the Stage-1 TTNN denoise, same device | normalised chunk PCC 0.999997 |
+    | Traced vs untraced, determinism | bit-identical; 5 trace replays bit-identical; 2 CQs == 1 CQ |
+    | Inference on p150a (warm, batch 1, median of 50) | **33.6 ms end-to-end** = host encode 1.0 + upload 1.2 + traces 31.1 (vision 8.7 · LLM 8.8 · adapter 4.9 · denoise 8.9 = 2.2 per step) + readback and decode 0.2 |
+    | Same tree with the Stage-1 TTNN denoise (`GR00T_DIT_BACKEND=ttnn`) | 40.4 ms end-to-end (denoise 16.0) → megakernel −16.8 % end-to-end, −44 % denoise |
+    | Served over HTTP by this image (warm, batch 1) | 32.5 ms device · 34.6 ms server-side incl. PNG decode |
+    | Same policy on an RTX 5090 (same host, NVIDIA's unmodified `Gr00tPolicy`, eager PyTorch bf16 + flash-attn 2) | 58.0 ms end-to-end (model 55.4) → p150a 1.73× faster |
 
     ### Caveats
 
-    - Stage-1 path: TTNN ops in four Metal traces, DiT matmul weights bfp8, everything else bf16. The megakernel denoise (weight streaming measured at 464 GB/s bf16 / 414 GB/s bfp8) is not in this image.
+    - Stage-2 path: one persistent megakernel runs the denoise (bfp8 weights streamed from a DRAM arena once per step, 110 cores, distributed LayerNorm, per-head flash SDPA); the rest is TTNN ops in three bf16 Metal traces. Its binaries (≈ 131.7 KB) fit 4.5 KB under the 136,192 B kernel-config ring, so the device runs with 64 KiB less L1 per core and the VL adapter's intermediates in DRAM (+0.4 ms); the DiT intermediates are not observable from the served model. `GR00T_DIT_BACKEND=ttnn` serves the Stage-1 denoise from the same image.
+    - Fidelity vs the Stage-1 path: gate outcomes equal on 8 of the 9 observations (the demo and 7 of the 8 multi-sample goldens); the one miss is `right_arm` on `t3_s300_seed2`, PCC 0.99987 vs the 0.9999 gate (the TTNN path passes it; the normalised chunk passes on both). The 16.8 % end-to-end gain is below the plan's 30 %.
     - One embodiment and layout: GR1 arms and hands, one `ego_view` camera, instruction ≤ 102 tokens; anything else is refused with 400. Batch 1.
-    - Noise is seeded by default (seed 0) so the output is reproducible; NVIDIA's `Gr00tPolicy` draws fresh noise per call. Outputs are the base checkpoint's actions for the GR1 sim data config.
+    - Noise is seeded by default (seed 0, the seed the fp32 reference was dumped with) so the output is reproducible; NVIDIA's `Gr00tPolicy` draws fresh noise per call. Outputs are the base checkpoint's actions for the GR1 sim data config.
     - Not an OpenAI-compatible API; `GET /v1/models` is a stub.
-    - First boot is cold (weight conversion + kernel JIT, READY after 1 min 43 s); later boots take 10–20 s. Validated on tt-metal `668c2907575`, single p150a only. Details: [`SERVING.md`](SERVING.md), [`GPU_COMPARISON.md`](GPU_COMPARISON.md).
+    - First boot is cold (weight conversion + kernel JIT incl. the megakernel, READY after 2 min 14 s); later boots take about 11 s. Validated on tt-metal `668c2907575`, single p150a only. Details: [`SERVING.md`](SERVING.md), [`GPU_COMPARISON.md`](GPU_COMPARISON.md).
 
     ### Licensing