Stage-2 release: DiT denoise as one persistent megakernel (snapshot 009de892, image 4cfc5e1ffe29)

Replaces the Stage-1 (TTNN-op denoise) package with the megakernel release: image sha256:4cfc5e1ffe299b02f05d09f2ac79a840690a135ec81ffa9da79cf99be87f6636 (tt-model/gr00t-n1.7-3b-p150:4cfc5e1ffe29), port snapshot 009de892c895 on tt-metal 668c290757, code/ digest 9efa5512d0634561. `tt-model push --publish` uploaded code/, image/, tt_kernel_manifest.json, requirements.lock and the generated card and pruned the code/ and image/ files this bundle no longer ships; this commit restores the reviewed staging card (front matter: license other / nvidia-license, pipeline_tag robotics, base_model nvidia/GR00T-N1.7-3B, 13 tags; body = the tt-model.yaml card with the simplified quickstart bullets) and the authored files tt-model.yaml, SERVING.md, GPU_COMPARISON.md, LICENSE, LICENSE-NOTICE.md, requirements.lock and media/*. Card text per docs/publish/review-mk.md.

GPU_COMPARISON.md

@@ -2,8 +2,9 @@
 
 Facts only. The GPU rows were measured on the build host during the port's golden study (2026-09-14,
 `docs/research/golden-multisample.md` §3 of the port project, "N1.7 GPU timing on an idle machine"); the p150a rows are the
-port's own Stage-1 benchmark and the served numbers of this image. Nothing here was re-measured for this file; every number
-names its source.
+port's own benchmark on the shipped snapshot (Stage 2: megakernel DiT denoise) plus, for reference, the same benchmark on the
+Stage-1 TTNN path of the same tree, and the served numbers of this image. Nothing here was re-measured for this file; every
+number names its source.
 
 ## What was run
 
@@ -16,7 +17,8 @@ names its source.
 | GPU software | Python 3.12.13, torch 2.9.0+cu128, transformers 4.57.3, flash-attn 2.8.3 — the `uv.lock` of the Isaac-GR00T checkout (`docs/research/ref-env-n17.md` §1) |
 | GPU loop | one process per configuration, 20 warm-up + 20 timed iterations, `torch.cuda.synchronize()` before and after every call, medians (`golden/multi_scripts/time_n17_idle.py`) |
 | GPU timing definitions | `get_action` = model only (backbone + action head); `e2e` = `get_action` + host preprocessing (albumentations chain, Qwen2-VL processor, tokenisation) + numpy decode; `backbone` and `action head` are the GPU sub-steps (`action head` = 4 DiT steps + encoders/decoder) |
-| p150a (device) | one Blackhole p150a, tt-metal `668c2907575`, the port's `benchmarks/bench_e2e.py` (Stage-1: TTNN ops in four Metal traces, `mixed_dit` = bfp8_b DiT matmul weights / bf16 elsewhere, `per_stage` layout, tuned Qwen3-VL tower, device-side text-embedding gather, 2 command queues): 10 warm-up calls, then **50 calls** of `encode → upload → 4 × execute_trace → one sync → D2H → decode`, host wall-clock medians; a second 50-call pass with a sync after every trace gives the per-trace split — `code/models/experimental/gr00t/benchmarks/results/bench_e2e_n17_snapshot.json` |
+| p150a (device, Stage 2 = this image's path) | one Blackhole p150a, tt-metal `668c2907575`, port snapshot `009de892c895`, the port's `benchmarks/bench_e2e.py` with its defaults (`dit_backend=megakernel`, bfp8_b arena, `mixed_dit`, `per_stage`, tuned Qwen3-VL tower, device-side text-embedding gather, 2 command queues, `worker_l1_size` 1,395,712): 10 warm-up calls, then **50 calls** of `encode → upload → 4 × execute_trace → one sync → D2H → decode`, host wall-clock medians; a second 50-call pass with a sync after every trace gives the per-trace split — `code/models/experimental/gr00t/benchmarks/results/bench_e2e_n17_20260918-050929.json` (idle host; the first run `…-050724.json`, taken while a CPU test suite was finishing, differs only in the host rows: e2e 65.8, `host_encode` 8.5) |
+| p150a (device, Stage 1 reference) | the same benchmark with `--dit-backend ttnn` on the same tree the same day (`bench_e2e_n17_20260918-032335.json`: the Stage-1 TTNN denoise, firmware-default L1, adapter intermediates in L1) — what the 2026-09-17 Stage-1 package served |
 | p150a (served) | this image, `POST /predict` on the demo observation, seed path, 50 requests after 5 warm-ups (`code/scripts/bench_http.py`), server-side `timing_ms` |
 
 ## Correctness reference (what "same output" means here)
@@ -26,18 +28,23 @@ flash-attention GPU path sits at a measurable distance from that golden, which b
 be asked to reproduce (gates: `code/models/experimental/gr00t/tests/tt/gates/gates_multi.json`, the `multi2x` rule over the seven goldens; the canonical
 thresholds in `common/golden_tap_map.json` `thresholds.n17` are its input):
 
-| tensor | official bf16 GPU vs fp32 golden (PCC) | p150a Stage-1 vs fp32 golden (PCC) | gate |
-|---|---:|---:|---:|
-| `vit_deepstack_mergers[0]` (first deepstack feature, 1792 × 2048) | 0.99995 | 0.999849 | 0.9998 |
-| `vl_self_attention_out` (DiT conditioning, 469 tokens × 2048) | 0.98592 | 0.97447 | 0.95 |
-| `action_pred_valid` (40 steps × 17 action dims, normalised) | 0.99996 | 0.99995 | 0.997 |
-| `unnormalized_actions` — `eef_9d` / `joint_position` | 0.99998 / 0.99997 | 0.99998 / 0.99996 | 0.9996 / 0.9998 |
-| `unnormalized_actions.gripper_position` (near-constant, max\|d\|) | 0.77 (PCC, for scale) | max\|d\| 0.0038 | abs ≤ 0.32 |
-
-p150a values: `tests/tt/results/test_e2e_{untraced_taps_vs_golden,predict_actions_vs_golden}_n17_20260917-2053*.json` (the
-snapshot validation run; identical to the live-tree run of 18:14 the same day). All 58 rows pass; on six additional DROID
-goldens five pass every tap and `t2_s120` misses `llm_layer_first` (0.98998 vs 0.991) and `joint_position` (PCC 0.99977 /
-max\|d\| 0.096 vs 0.9998 / 0.083) — see the card's Accuracy row and `docs/plan/n17-fidelity.md` of the port project.
+| tensor | official bf16 GPU vs fp32 golden (PCC) | p150a Stage 2 (this image) vs fp32 golden (PCC) | p150a Stage 1 (2026-09-17 package) | gate |
+|---|---:|---:|---:|---:|
+| `vit_deepstack_mergers[0]` (first deepstack feature, 1792 × 2048) | 0.99995 | 0.999849 | 0.999849 | 0.9998 |
+| `vl_self_attention_out` (DiT conditioning, 469 tokens × 2048) | 0.98592 | 0.97447 | 0.97447 | 0.95 |
+| `action_pred_valid` (40 steps × 17 action dims, normalised) | 0.99996 | 0.99995 | 0.99995 | 0.997 |
+| `unnormalized_actions` — `eef_9d` / `joint_position` | 0.99998 / 0.99997 | 0.99998 / 0.99996 | 0.99998 / 0.99996 | 0.9996 / 0.9998 |
+| `unnormalized_actions.gripper_position` (near-constant, max\|d\|) | 0.77 (PCC, for scale) | max\|d\| 0.0031 | max\|d\| 0.0038 | abs ≤ 0.32 |
+| `action_pred_normalized`, megakernel vs the Stage-1 TTNN denoise on the same device | — | ≥ 0.999992 (7 DROID samples) | — | 0.9999 (oracle gate) |
+
+p150a Stage-2 values: `tests/tt/results/test_e2e_{untraced_taps_vs_golden,predict_actions_vs_golden}_n17_20260918-0452*.json` (the snapshot
+validation run, commit `009de892c895`; bit-equal to the live tree's megakernel-default run of 04:11 the same day); the backbone / adapter taps
+are bit-identical to the Stage-1 package (the megakernel does not touch them), the head taps move by ≤ 1e-5 in PCC. All 30 rows pass
+(27 gated taps — the DiT's per-step internals are not observable inside one `generic_op`, so the Stage-1 path's 58-row comparison shrinks to
+30 rows). On six additional DROID goldens the actions pass on five: `t2_s120` misses `llm_layer_first` (0.98998 vs 0.991) and
+`joint_position` (PCC 0.99976 / max\|d\| 0.107 vs 0.9998 / 0.083) as it did on the Stage-1 path; `t1_s120` misses one megakernel-only
+per-dim gate (`pred_velocity[k=2]` dim 9: 0.99588 vs 0.996; its actions pass) — see the card's Accuracy row, `SERVING.md` §4 and
+`tests/tt/results/mk_default_summary.md` §3 of the port.
 
 ## GPU latency (RTX 5090, batch 1, medians of 20 iterations, ms)
 
@@ -57,38 +64,43 @@ RTX Pro 6000 TensorRT full pipeline 27.9 ms** (H100: backbone 8.8, head 12.3); t
 
 ## p150a latency (batch 1, ms)
 
-| row | median | p90 | source |
-|---|---:|---:|---|
-| `bench_e2e` end-to-end (host encode + upload + 4 traces + D2H + host decode) | **75.8** | 81.6 | `bench_e2e_n17_snapshot.json` `stages.e2e` |
-| device = upload + traces + D2H | 69.4 | 69.8 | `stages.device` |
-| 4 traces (one sync) | 66.1 | — | `stages.traces` |
-| trace `vision` (tuned Qwen3-VL tower: 24 blocks on 4 × 448 patches, 3 deepstack mergers + merger) | 17.5 | — | `stages.trace/vision` (split pass) |
-| trace `llm` (Qwen3 16 layers with deepstack injection, S 469 → 512) | 12.4 | — | `stages.trace/llm` |
-| trace `adapter` (VL-LN + 4 VL self-attention blocks, K/V hoist, state encoder) | 5.7 | — | `stages.trace/adapter` |
-| trace `denoise` (4 Euler steps × 32 DiT blocks + decoder) | 30.7 | — | `stages.trace/denoise` |
-| upload / D2H / host encode / host decode | 3.2 / 0.08 / 4.9 / 1.2 | — | `stages.*` |
-| served `timing_ms.device` (this image, 50 requests after 5 warm-ups) | 69.5 | 69.7 | `bench_http.py` run of the container validation (`docs/publish/build-n17.md`) |
-| served `timing_ms.total` (handler wall incl. base64 + four PNG decodes + host encode) | 84.4 | 94.5 | same |
-| served client wall (localhost, incl. HTTP + JSON) | 87.4 | 98.0 | same |
+| row | Stage 2 (this image) median | p90 | Stage 1 TTNN path, same tree (reference) | source (`stages.*` of the two bench JSONs) |
+|---|---:|---:|---:|---|
+| `bench_e2e` end-to-end (host encode + upload + 4 traces + D2H + host decode) | **62.8** | 68.6 | 79.7 (p90 91.0) | `e2e` |
+| device = upload + traces + D2H | 55.7 | 56.1 | 69.8 | `device` |
+| 4 traces (one sync) | 52.4 | — | 66.1 | `traces` |
+| trace `vision` (tuned Qwen3-VL tower: 24 blocks on 4 × 448 patches, 3 deepstack mergers + merger) | 17.5 | — | 17.5 | `trace/vision` (split pass) |
+| trace `llm` (Qwen3 16 layers with deepstack injection, S 469 → 512) | 12.4 | — | 12.4 | `trace/llm` |
+| trace `adapter` (VL-LN + 4 VL self-attention blocks, K/V hoist, state encoder; intermediates in DRAM under the megakernel) | 6.2 | — | 5.7 | `trace/adapter` |
+| trace `denoise` (4 Euler steps × 32 DiT blocks + decoder) | **16.5** (4.12 per step) | — | 30.7 | `trace/denoise` |
+| upload / D2H / host encode / host decode | 3.2 / 0.09 / 5.6 / 1.2 | — | 3.7 / 0.09 / 8.6 / 1.2 | `upload`, `d2h`, `host_encode`, `host_decode` |
+| served `timing_ms.device` (this image, 50 requests after 5 warm-ups) | 55.9 | 56.3 | 69.5 (the Stage-1 image) | `bench_http.py` run of the container validation (`docs/publish/build-mk-n17.md`) |
+| served `timing_ms.total` (handler wall incl. base64 + four PNG decodes + host encode) | 69.1 | 72.7 | 84.4 | same |
+| served client wall (localhost, incl. HTTP + JSON) | 72.4 | 76.0 | 87.4 | same |
+
+The Stage-1 column is `bench_e2e_n17_20260918-032335.json` (`--dit-backend ttnn`, same tree, same day; its `host_encode` row was taken on a
+busier host) and, for the served rows, the 2026-09-17 Stage-1 package (`docs/publish/build-n17.md`). Stage 2 vs Stage 1: device −20 %,
+denoise trace −46 %, e2e −21 % here and −19 % in the live tree's same-session pair (64.4 vs 79.7 ms, `tests/tt/results/mk_default_summary.md`
+§5.2); the adapter trace costs +0.6 ms for its DRAM intermediates. The plan's −30 % e2e target was not reached: the denoise is now 31 % of
+the traces and the backbone (vision + LLM, 29.9 ms) is the largest block.
 
 ## Comparison with matching definitions
 
 Ratio = GPU ms / p150a ms (> 1 means the p150a is faster).
 
-| row | p150a | GPU | ratio |
-|---|---:|---:|---:|
-| end-to-end policy call: `bench_e2e` e2e (encode + device + decode) vs eager bf16 + FA2 e2e incl. host pre/post (`A2`) | 75.8 | 76.45 | **1.01** |
-| model only: p150a device (upload + traces + D2H) vs GPU `get_action` (`A2`) | 69.4 | 63.19 | **0.91** |
-| denoise: p150a `denoise` trace vs GPU action head (`A2`) | 30.7 | 33.71 | see rows above |
-| vision + LLM: p150a `vision` + `llm` traces vs GPU backbone (`A2`) | 17.5 + 12.4 | 29.43 | see rows above |
-| end-to-end vs NVIDIA's published H100 TensorRT full pipeline (one camera, different hardware and input) | 75.8 | 27.9 | GPU faster |
-
-Reading: against NVIDIA's eager bf16 path on the same host the Stage-1 p150a port is at or slightly under parity end-to-end;
-the DROID layout's four 16:9 frames make the vision tower the largest p150a stage (the GR1 packages' towers are 9–12 ms),
-and the p150a denoise (four Euler steps over 32 blocks) is the part the persistent megakernel (Stage 2, not in this image;
-weight streaming measured at 464 GB/s bf16 / 414 GB/s bfp8 in `tests/tt/results/mk_k1_summary.md`) is meant to replace. The
-GPU's host pre/post (≈ 13 ms in the `A2` run) and the p150a's `host encode` + `host decode` are the same kind of host work;
-HTTP/JSON framing is excluded on both sides except in the "served client wall" row.
+| row | p150a (Stage 2) | GPU | ratio | Stage-1 ratio (2026-09-17 package) |
+|---|---:|---:|---:|---:|
+| end-to-end policy call: `bench_e2e` e2e (encode + device + decode) vs eager bf16 + FA2 e2e incl. host pre/post (`A2`) | 62.8 | 76.45 | **1.22** | 1.01 |
+| model only: p150a device (upload + traces + D2H) vs GPU `get_action` (`A2`) | 55.7 | 63.19 | **1.13** | 0.91 |
+| denoise: p150a `denoise` trace (the megakernel) vs GPU action head (`A2`) | 16.5 | 33.71 | **2.04** | 1.10 |
+| vision + LLM: p150a `vision` + `llm` traces vs GPU backbone (`A2`) | 17.5 + 12.4 = 29.9 | 29.43 | 0.98 | 0.98 |
+| end-to-end vs NVIDIA's published H100 TensorRT full pipeline (one camera, different hardware and input) | 62.8 | 27.9 | GPU faster | GPU faster |
+
+Reading: with the DiT denoise as one persistent megakernel the p150a is now ahead of NVIDIA's eager bf16 path on the same host both
+end-to-end (1.22×) and model-only (1.13×), and its denoise runs in half the time of the GPU's eager action head; the backbone traces are at
+parity with the GPU backbone, and the DROID layout's four 16:9 frames make the vision tower the largest p150a stage (the GR1 packages'
+towers are 9–12 ms). The GPU's host pre/post (≈ 13 ms in the `A2` run) and the p150a's `host encode` + `host decode` are the same kind of
+host work; HTTP/JSON framing is excluded on both sides except in the "served client wall" row.
 
 Not measured / not claimed: p150a power (no power or efficiency comparison); `torch.compile`, TensorRT or FP8 on the GPU for
 N1.7 on this host; the other N1.7 layouts (G1 4:3, three-camera). GPU numbers were not re-run for this card; they are copied

LICENSE-NOTICE.md

@@ -9,7 +9,7 @@ and a card. It contains **no model weights**. Five licences apply to what is her
 | Vendored tokenizer / image-processor files `code/gr00t_p150/assets/tokenizer/n17/{vocab.json, merges.txt, tokenizer_config.json, preprocessor_config.json}` | in this repo and in the image (`/opt/tt-metal/gr00t_p150/assets/tokenizer/n17`) | **byte-identical** (sha256 in `code/gr00t_p150/assets/tokenizer/n17/PROVENANCE.md`) to the same-named files of [`Qwen/Qwen3-VL-2B-Instruct`](https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct) @ `89644892e4d85e24eaac8bacfd4f463576704203`, **Apache License 2.0**. GR00T N1.7's backbone (`nvidia/Cosmos-Reason2-2B`) is a fine-tune of that model and ships the same tokenizer / processor files; they are not part of the weights repo, which is why they ship here. Verified: loading them reproduces the golden N1.7 `input_ids`, `pixel_values` and `state` bit-exactly. |
 | Demo observation `media/{exterior_image_1_left,wrist_image_left}_{tm15,t0}.png`, `media/demo_frames.png`, `media/demo_observation.json`, `code/gr00t_p150/demo/n17/` (four frames, raw state, instruction, the reference initial noise and the fp32 / bf16 reference actions) | in this repo and in the image | one step of the DROID sample episode distributed with the [Isaac-GR00T](https://github.com/NVIDIA/Isaac-GR00T) repository (`demo_data/droid_sample`, episode 1, step 40; frames at t-15 and t of both 320×180 cameras); DROID itself is released by its authors under [CC-BY-4.0](https://droid-dataset.github.io/). The reference actions were computed with NVIDIA's unmodified code on the pinned weights and are therefore derivative of the weights (NVIDIA License §3.2 applies to their use). |
 | Port and serving code: `code/models/experimental/gr00t/` (the tt-nn port of GR00T N1.5 / N1.6 / N1.7: `common/`, `reference/`, `tt/` incl. the tuned Qwen3-VL tower `tt/qwen3vl_vision.py` and `tt/megakernel/`, `tests/`, `benchmarks/`), `code/gr00t_p150/` (FastAPI server, schemas, smoke test, demo builder), `code/scripts/` | in this repo and in the image (`/opt/tt-metal/...`) | **Apache-2.0** (`SPDX-License-Identifier: Apache-2.0`, © 2026 Tenstorrent USA, Inc. in every source file). Because the port only runs the NVIDIA-licensed weights and is distributed together with the reference actions, the package as a whole is offered under the NVIDIA License's non-commercial use limitation; the Apache-2.0 grant on the code itself is unaffected. |
-| tt-metal / tt-nn: `code/models/common/utility_functions.py`, `code/models/tt_dit/**` (incl. the verbatim Qwen3-VL tower `encoders/qwen3vl/vision_qwen3vl.py`, the oracle the tuned tower is checked against), `code/models/demos/deepseek_v3_b1/unified_kernel_descriptor.py` and the whole tt-metal build inside the image (`/opt/tt-metal`, commit `668c290757550588d0ce46b180c344a462a2aaf5`) | in this repo (the 16 files) and in the image | **Apache-2.0** ([tenstorrent/tt-metal](https://github.com/tenstorrent/tt-metal/blob/main/LICENSE)). |
+| tt-metal / tt-nn: `code/models/common/utility_functions.py`, `code/models/tt_dit/**` (incl. the verbatim Qwen3-VL tower `encoders/qwen3vl/vision_qwen3vl.py`, the oracle the tuned tower is checked against), `code/models/demos/deepseek_v3_b1/unified_kernel_descriptor.py` and `unified_kernels/{kernel_op_api,kernel_utils,dataflow_utils,mcast}.hpp` (the megakernel's `generic_op` descriptor and the device-kernel headers it #includes) and the whole tt-metal build inside the image (`/opt/tt-metal`, commit `668c290757550588d0ce46b180c344a462a2aaf5`) | in this repo (the 20 files) and in the image | **Apache-2.0** ([tenstorrent/tt-metal](https://github.com/tenstorrent/tt-metal/blob/main/LICENSE)). |
 | Third-party Python packages inside the image (`requirements.lock`: torch 2.11.0+cpu, torchvision 0.26.0+cpu, transformers 5.12.1, opencv-python-headless 4.8.1.78, numpy 1.26.4, fastapi, uvicorn, pydantic, pillow, safetensors, huggingface_hub, loguru, ...) and the Ubuntu 22.04 base | in the image | their respective licences (BSD-3 / Apache-2.0 / MIT / HPND; Ubuntu package licences). |
 
 ## Practical reading

README.md

@@ -4,11 +4,24 @@ tags:
 - p150
 - tt-dit-server
 - tt-model-cache
-- tt-model-catalog
 - tt-model-container
+- tenstorrent
+- ttnn
+- tt-metal
+- tt-nn
+- robotics
+- vla
+- gr00t
+- tt-model-catalog
+pipeline_tag: robotics
+license: other
+license_name: nvidia-license
+license_link: https://huggingface.co/nvidia/GR00T-N1.7-3B/blob/main/LICENSE
+base_model:
+- nvidia/GR00T-N1.7-3B
 ---
 
-# gr00t-n1.7-3b-p150
+# GR00T-N1.7-3B-p150
 
 NVIDIA Isaac GR00T N1.7 (3B vision-language-action policy) running on one Tenstorrent Blackhole p150a via tt-nn, in the DROID layout: two cameras × two frames (t−15 and t) at 16:9, a 17-dim Franka state and a task instruction in, a 40-step chunk of end-effector, gripper and joint actions out. Stage-2 release: the whole DiT denoise runs as one persistent megakernel streaming bfp8 weights from a DRAM arena; vision (tuned Qwen3-VL tower), LLM and adapter are TTNN ops in Metal traces; `GR00T_DIT_BACKEND=ttnn` selects the Stage-1 denoise.
 Weights: [nvidia/GR00T-N1.7-3B](https://huggingface.co/nvidia/GR00T-N1.7-3B) · Paper: [arXiv:2503.14734](https://arxiv.org/abs/2503.14734) · Upstream code: [NVIDIA/Isaac-GR00T](https://github.com/NVIDIA/Isaac-GR00T) (`main`, the `n1.7-release` line) · Port: `code/models/experimental/gr00t` and the server `code/gr00t_p150` in this repo
@@ -24,7 +37,8 @@ tt-model pull  changh95/GR00T-N1.7-3B-p150 --with-weights
 tt-model serve changh95/GR00T-N1.7-3B-p150
 ```
 
-`pull --with-weights` downloads the Docker image and the [`nvidia/GR00T-N1.7-3B`](https://huggingface.co/nvidia/GR00T-N1.7-3B) weights at `2fc962b973bccdd5d8ce4f67cc63b264d6886495` (into your HF cache; they are not in the image). `serve` starts the model's own HTTP server on port 20000 (or the next free port, if that one is busy); the first start compiles kernels for your device, which takes several minutes, and the server is ready when it logs `Application startup complete`.
+- Weights [`nvidia/GR00T-N1.7-3B`](https://huggingface.co/nvidia/GR00T-N1.7-3B) at `2fc962b973bc` (6.91 GB) go to your HF cache; the image does not contain them.
+- Serves on port 20000 (or the next free port); ready when the log says `Application startup complete`.
 
 ### Run with tt-cli
 
@@ -100,4 +114,3 @@ The exact sources the image was built from — `code/` in this repo is byte-iden
 | tt-metal | [`668c290757550588d0ce46b180c344a462a2aaf5`](https://github.com/tenstorrent/tt-metal/commit/668c290757550588d0ce46b180c344a462a2aaf5) |
 | `code/` digest | `9efa5512d0634561` (sha256, first 16 hex digits) |
 | built | 2026-09-17T20:42:14+00:00 by tt-model 0.1.0 |
-

SERVING.md

@@ -4,17 +4,18 @@ This repo is a **tt-model container package source**: `tt-model.yaml` + the code
 --container` builds the OCI image (tt-metal from source + the port + the HTTP stack), `tt-model serve` runs it against
 the card, `tt-model push` publishes it here. The server is `code/gr00t_p150/server/app.py` (FastAPI, `kind:
 tt-dit-server`, shared with the N1.5 / N1.6 packages); the device model is the tt-nn port `code/models/experimental/gr00t`
-(Stage-1 path: TTNN ops captured in four Metal traces, DiT matmul weights bfp8_b, everything else bf16, the port's tuned
-Qwen3-VL vision tower). Weights are a pinned pointer, never in the image.
+(**Stage-2 path**: the whole DiT denoise as one persistent megakernel `generic_op` streaming a bfp8_b DRAM weight arena, inside
+the fourth Metal trace; vision / LLM / adapter as TTNN ops in the other three traces; DiT matmul weights bfp8_b, everything
+else bf16; the port's tuned Qwen3-VL vision tower). Weights are a pinned pointer, never in the image.
 
 | | |
 |---|---|
 | tt-metal | main `668c290757550588d0ce46b180c344a462a2aaf5` (`v0.79.0-dev20260914-1-g668c290757`), built from source inside the image; torch pin 2.11.0 (+cpu in the image) |
 | weights | `nvidia/GR00T-N1.7-3B` @ `2fc962b973bccdd5d8ce4f67cc63b264d6886495` — `model-0000{1,2}-of-00002.safetensors` (6,910,361,856 B of bf16 tensors, 1,031 tensors), `model.safetensors.index.json`, `config.json`, `processor_config.json`, `statistics.json`, `embodiment_id.json`, `LICENSE` (NVIDIA License, non-commercial) |
-| app | `gr00t_p150.server.app:app` (uvicorn `--lifespan on`; the lifespan resolves the snapshot, opens the device, builds `Gr00tTT`, captures the traces, runs `GR00T_WARMUP_RUNS` traced predictions and checks them against the shipped fp32 golden before READY) |
-| device recipe | `models.experimental.gr00t.tt.device.open_gr00t_device(device_id, trace_region_size=64 MiB, l1_small_size=32768, num_command_queues=1)` — the port's validated parameters; asserts the 11×10 compute / 8×1 DRAM grid of a p150. (The port's own benchmark opens 2 command queues and overlaps the small input writes with the vision trace; the server keeps 1 queue, worth ≈ 0.4 ms per call — `docs/plan/n17-perf.md` §4.2 of the port project.) |
-| port source | `code/models/experimental/gr00t` = the frozen snapshot commit `5833d4befc8e748a4b8342dc2329b09d36d163f8` (ref `gr00t-p150-snapshot-n17`, parent = `668c2907575`) of the tt-metal working tree `models/experimental/gr00t`; `code/gr00t_p150` (server) and `code/scripts` were written for these releases |
-| only served configuration | version `n17`, embodiment `oxe_droid_relative_eef_relative_joint` (slot 24), static layout `droid` (2 cameras × 2 frames = 4 images × 112 tokens on a 16×28-patch grid, `L_max` 51 text tokens, LLM sequence padded to 512), `mixed_dit` dtype policy, `per_stage` trace layout, tuned tower (`GR00T_VIT_FORK=1`), batch 1 |
+| app | `gr00t_p150.server.app:app` (uvicorn `--lifespan on`; the lifespan resolves the snapshot, resolves the `TTPolicy` from `GR00T_*`, opens the device the policy needs, builds `Gr00tTT`, captures the traces, runs `GR00T_WARMUP_RUNS` traced predictions and checks them against the shipped fp32 golden before READY) |
+| device recipe | `models.experimental.gr00t.tt.model.open_model_device(policy, trace_region_size=64 MiB, l1_small_size=32768, num_command_queues=2, device_id)`: under the megakernel backend the device is opened with `worker_l1_size` 1,461,248 − 65,536 = **1,395,712 B** (the 64 KiB cut buys the 136,192 B kernel-config ring the megakernel binaries need) and the VL adapter's intermediates live in DRAM; 2 command queues (per-call input writes on CQ 1 overlap the traces, bit-equal to 1 CQ — `tests/tt/test_mk_2cq.py`); asserts the 11×10 compute / 8×1 DRAM grid of a p150. `GR00T_DIT_BACKEND=ttnn` opens the firmware-default L1 and serves the Stage-1 TTNN denoise |
+| port source | `code/models/experimental/gr00t` = the frozen snapshot commit `009de892c895917ad6abe9a6c36518f00316ed4b` (ref `gr00t-p150-snapshot-mk`, parent = `668c2907575`) of the tt-metal working tree `models/experimental/gr00t`; the `tt/megakernel/` tree is the K4b checkpoint (16 files, digest `7df61c35e9dc7857`, `tests/tt/results/mk_tree_20260918-041439.sha256`, re-hashed by `verify:` inside the image); `code/gr00t_p150` (server) and `code/scripts` were written for these releases |
+| only served configuration | version `n17`, embodiment `oxe_droid_relative_eef_relative_joint` (slot 24), static layout `droid` (2 cameras × 2 frames = 4 images × 112 tokens on a 16×28-patch grid, `L_max` 51 text tokens, LLM sequence padded to 512), `mixed_dit` dtype policy, `per_stage` trace layout, `dit_backend=megakernel` with the `bfp8_b` arena, 2 command queues, tuned tower (`GR00T_VIT_FORK=1`), batch 1 |
 
 Below, `$ROOT` = the tt-models tooling checkout (`/home/deepgadget/experiments/tt-models` on the build host: `tt-model` CLI in
 `$ROOT/.venv/bin`, rootless Docker env in `$ROOT/bin/docker-env.sh`), `$TREE` = the tt-metal tree with the built `python_env`
@@ -33,11 +34,13 @@ media/{exterior_image_1_left,wrist_image_left}_{tm15,t0}.png   the four DROID de
 media/demo_observation.json   the matching raw state + instruction, media/demo_actions_vs_golden.png the served chunk vs the fp32 reference
 code/models/experimental/gr00t/    the port: common/ (host preprocessing incl. the DROID eval chain + Qwen2-VL patching, prompts, normalisation,
                                    checkpoint, weight plan), reference/ (fp32 torch reference), tt/ (device model, layers, traces, the tuned
-                                   Qwen3-VL tower tt/qwen3vl_vision.py, tt/megakernel/ Stage-2 work), tests/ (CPU + device suites, cited results),
-                                   benchmarks/ (bench_e2e etc. + results)
-code/models/common/utility_functions.py, code/models/tt_dit/{utils,layers,parallel,encoders/qwen3vl}/..., code/models/demos/deepseek_v3_b1/unified_kernel_descriptor.py
-                                   the tt-metal files the port imports from the tree other than itself (staged by tt-model from `source.code`;
-                                   encoders/qwen3vl/vision_qwen3vl.py is the verbatim tower = the oracle behind GR00T_VIT_FORK=0)
+                                   Qwen3-VL tower tt/qwen3vl_vision.py, tt/action_head.py with both DiT backends, tt/megakernel/ = the persistent
+                                   DiT megakernel: arena.py (DRAM weight arena), core_map.py, descriptors.py, dit_program.py, kernels/dit_kernel.cpp +
+                                   kernels/ops/*.hpp), tests/ (CPU + device suites, the cited result JSONs and the WP summaries), benchmarks/ (+ results)
+code/models/common/utility_functions.py, code/models/tt_dit/{utils,layers,parallel,encoders/qwen3vl}/..., code/models/demos/deepseek_v3_b1/{unified_kernel_descriptor.py, unified_kernels/{kernel_op_api,kernel_utils,dataflow_utils,mcast}.hpp}
+                                   the 20 tt-metal files the port imports or #includes from the tree other than itself (staged by tt-model from `source.code`;
+                                   unified_kernel_descriptor.py builds the megakernel's generic_op and dit_kernel.cpp #includes the four unified_kernels headers;
+                                   vision_qwen3vl.py is the verbatim tower = the oracle behind GR00T_VIT_FORK=0)
 code/gr00t_p150/              server/{app.py, schemas.py, smoke_test.py}, demo/n17/ (canonical DROID demo observation, reference noise + golden),
                               assets/tokenizer/n17/ (vendored Qwen2 tokenizer + Qwen2-VL preprocessor_config.json, PROVENANCE.md)
 code/scripts/                 download_weights.sh (pinned snapshot -> HF cache), bench_http.py (HTTP latency, 50 requests)
@@ -68,14 +71,16 @@ export TT_WEIGHTS_REVISION=2fc962b973bccdd5d8ce4f67cc63b264d6886495      # scrip
 export TT_MESH_SHAPE=1x1 TT_DEVICE_ID=0 TT_METAL_VISIBLE_DEVICES=0
 export GR00T_VERSION=n17 GR00T_EMBODIMENT=oxe_droid_relative_eef_relative_joint GR00T_LAYOUT=droid
 export GR00T_POLICY=mixed_dit GR00T_TRACE_LAYOUT=per_stage GR00T_VIT_FORK=1
+export GR00T_DIT_BACKEND=megakernel GR00T_MK_ARENA_DTYPE=auto GR00T_NUM_CQS=2   # = the port's defaults; GR00T_DIT_BACKEND=ttnn for the Stage-1 A/B
 export GR00T_TOKENIZER_DIR=$PWD/code/gr00t_p150/assets/tokenizer/n17
-export GR00T_TT_CACHE=$HOME/.cache/gr00t-tt                              # host .pt plan tier + .tensorbin device tier (~8.4 GB for n17)
+export GR00T_TT_CACHE=$HOME/.cache/gr00t-tt                              # host .pt plan tier + .tensorbin device tier (~2.9 GB for n17 under the megakernel)
 export GR00T_WARMUP_RUNS=5
 
 # import check, no device (what the image's verify.sh does):
 $TREE/python_env/bin/python -c "import gr00t_p150.server.app as a; assert a.app; print('ok')"
 
-# serve (opens the device, loads the weights, captures the four traces, 5 warm-up predictions checked against the golden, then READY).
+# serve (resolves the policy, opens the device with the megakernel's L1 cut, loads the weights, builds the arena, captures the
+# four traces, 5 warm-up predictions checked against the golden, then READY).
 # The one Blackhole card of the build host is shared with other agents: run server + client + shutdown inside ONE lock hold.
 DEVICE_LOCK_TIMEOUT=14400 /home/deepgadget/experiments/gr00t/bin/with-device.sh bash -c '
   $TREE/python_env/bin/python -m uvicorn --host 127.0.0.1 --port 20017 --lifespan on gr00t_p150.server.app:app & UV=$!
@@ -85,21 +90,26 @@ DEVICE_LOCK_TIMEOUT=14400 /home/deepgadget/experiments/gr00t/bin/with-device.sh
 ```
 
 Boot log landmarks (they drive `tt-model serve`'s boot checklist): `Loading weights: nvidia/GR00T-N1.7-3B @ 2fc962b9…` →
-`Tokenizer files: …` → `Opening device 0 (trace_region_size=67108864, l1_small_size=32768)` → `Loading pipeline: Gr00tTT.from_pretrained(...)`
-→ `Model built in N s (1063 tensors, 4040 MB on device, cache path warm|cold)` → `Warming up: capture per_stage traces on the n17 demo
-observation, then 5 traced predict(s)` → `Warmup k/5: … ms` → `Warm-up fidelity vs golden fp32: actions PCC 0.999962…` → `Warmup complete (…)`
-→ uvicorn `Application startup complete`. Startup failures raise and uvicorn exits non-zero (no CPU fallback); a warm-up PCC below
-`GR00T_WARMUP_PCC_MIN` (0.999) fails the boot. Stop with SIGTERM / Ctrl-C: the lifespan releases the traces and device tensors and closes
-the device (`Releasing traces and device tensors`, `Closing device`).
-
-Expected smoke line on the demo observation (the shipped image, cold boot cycle of the container validation,
-`publish/logs/smoke-n17-c1-20260917-212555.log` of the build host; the host run gives the same fidelity):
+`Tokenizer files: …` → `Opening device 0 (trace_region_size=67108864, l1_small_size=32768, num_command_queues=2, dit_backend=megakernel,
+mk_arena_dtype=bfp8_b)` → `Loading pipeline: Gr00tTT.from_pretrained(...)` → `Model built in N s (weights … s, 529 tensors, 3008 MB on device,
+cache path warm|cold; dit_backend megakernel, arena bfp8_b, adapter_mem DRAM, head {mk_arena_plan_pack_s …, mk_upload_s …}; ttnn DiT weights
+skipped: 534 tensors / 1032 MB)` → `Warming up: capture per_stage traces on the n17 demo observation, then 5 traced predict(s)` →
+`Warmup k/5: … ms` → `Warm-up fidelity vs golden fp32: actions PCC 0.999959…` → `Warmup complete (…)` → uvicorn `Application startup
+complete`. Under the megakernel the ttnn DiT weights are not uploaded (534 tensors / 1,032 MB skipped, `GET /info -> stage.weights`); the
+megakernel's own bfp8_b arena (~1 GB) is packed on the host and uploaded during the model build (`head_timing_s`). Startup failures raise
+and uvicorn exits non-zero (no CPU fallback); a warm-up PCC below `GR00T_WARMUP_PCC_MIN` (0.999) fails the boot. Stop with SIGTERM / Ctrl-C:
+the lifespan releases the traces and device tensors and closes the device (`Releasing traces and device tensors`, `Closing device`).
+
+Expected smoke line on the demo observation (the shipped image, cold-boot cycle of the container validation,
+`publish/logs/smoke-mk-n17-c1-<stamp>.log` of the build host; the host run of `tests/tt/results/mk_default_summary.md` §7 gives the same fidelity):
 
 ```
-PASS GR00T-N1.7-3B-p150 n17: PCC(actions)=0.999962 max|d|=0.0342 PCC(action_pred_valid)=0.999951 repeat_maxdiff=0.0e+00 seed_path_ok device_ms=70.19/70.04 total_ms=94.24/88.02 wall_ms=100 per_group[eef_9d=0.99998/0.0118 gripper_position=0.76234/0.0038 joint_position=0.99996/0.0342]
+PASS GR00T-N1.7-3B-p150 n17: PCC(actions)=0.999959 max|d|=0.0282 PCC(action_pred_valid)=0.999946 repeat_maxdiff=0.0e+00 seed_path_ok device_ms=56.51/56.35 total_ms=74.63/69.67 wall_ms=81 per_group[eef_9d=0.99998/0.0121 gripper_position=0.81268/0.0031 joint_position=0.99996/0.0282]
 ```
 
-Host boot with warm `GR00T_TT_CACHE` + `TT_METAL_CACHE`: model built ~1 s, capture ~2 s, 5 warm-ups ~76 ms each, READY after 6–8 s.
+Host boot on the build host with a **fresh, empty** `TT_METAL_CACHE` and warm weight tiers (`mk_default_summary.md` §7,
+`publish/logs/server_n17_20260918-043407.log`): `Model built in 9.5 s`, capture (JIT of every op + the megakernel) 39.3 s, boot to READY 52.7 s;
+smoke `device_ms=56.72/56.18`, `total_ms=71.69/67.9`. With warm kernel and weight caches the host boot is under 10 s.
 Offline overrides: `GR00T_WEIGHTS_DIR=<dir with the seven snapshot files>` (shimmed into a private hub cache so the port's
 `configs.snapshot_dir("n17")` resolves to it); `GR00T_TOKENIZER_DIR=<dir>` (must also hold `preprocessor_config.json`);
 `GR00T_TORCH_THREADS=<n>` (torch intra-op threads for the host preprocessing; unset in the image). Every variable is read in the
@@ -133,21 +143,21 @@ $ROOT/.venv/bin/tt-model push /home/deepgadget/experiments/gr00t/publish/build/g
 -v ~/.cache/tt-model/gr00t-n1.7-3b-p150/{cache,weights,tensors}:/{cache,weight-cache,tensor-cache} -p 20000:20000` with exactly
 `HF_MODEL=nvidia/GR00T-N1.7-3B`, `MESH_DEVICE=P150`, `TT_MESH_SHAPE=1x1`, `HF_HOME=/hf`, `TT_METAL_CACHE=/cache`,
 `TT_DIT_CACHE_DIR=/weight-cache`, `TT_CACHE_PATH=/tensor-cache` and the `serve.env` block of `tt-model.yaml`
-(`TT_WEIGHTS_REVISION`, `GR00T_VERSION=n17`, `GR00T_LAYOUT=droid`, `GR00T_VIT_FORK=1`, `GR00T_TT_CACHE=/weight-cache/gr00t-tt`,
-`GR00T_TOKENIZER_DIR=/opt/tt-metal/gr00t_p150/assets/tokenizer/n17`, …), and waits ≤ 1800 s for `Application startup complete`.
-Serves on 20000 or the next free port (printed). tt-cli users after the push: `tt serve changh95/GR00T-N1.7-3B-p150` /
-`tt model stop changh95/GR00T-N1.7-3B-p150`.
+(`TT_WEIGHTS_REVISION`, `GR00T_VERSION=n17`, `GR00T_LAYOUT=droid`, `GR00T_DIT_BACKEND=megakernel`, `GR00T_MK_ARENA_DTYPE=auto`,
+`GR00T_NUM_CQS=2`, `GR00T_VIT_FORK=1`, `GR00T_TT_CACHE=/weight-cache/gr00t-tt`, `GR00T_TOKENIZER_DIR=/opt/tt-metal/gr00t_p150/assets/tokenizer/n17`, …),
+and waits ≤ 1800 s for `Application startup complete`. Serves on 20000 or the next free port (printed). tt-cli users after the push:
+`tt serve changh95/GR00T-N1.7-3B-p150` / `tt model stop changh95/GR00T-N1.7-3B-p150`.
 
-Caches on the host (persist across boots): `~/.cache/tt-model/gr00t-n1.7-3b-p150/cache` (JIT kernels, `TT_METAL_CACHE`),
-`~/.cache/tt-model/gr00t-n1.7-3b-p150/weights/gr00t-tt/n17/...` (the port's host plan tier + `.tensorbin` device tier, ~8.4 GB, written on
-the first boot), weights in `~/.cache/huggingface/hub/models--nvidia--GR00T-N1.7-3B`.
+Caches on the host (persist across boots): `~/.cache/tt-model/gr00t-n1.7-3b-p150/cache` (JIT kernels incl. the megakernel, `TT_METAL_CACHE`),
+`~/.cache/tt-model/gr00t-n1.7-3b-p150/weights/gr00t-tt/n17/...` (the port's host plan tier + `.tensorbin` device tier, 2.9 GB under the megakernel — the skipped ttnn DiT tensors are not
+converted either; 8.4 GB on the Stage-1 path — written on the first boot), weights in `~/.cache/huggingface/hub/models--nvidia--GR00T-N1.7-3B`.
 
-Measured on the build host with this image (`docs/publish/build-n17.md` of the port project has every command and log):
+Measured on the build host with this image (`docs/publish/build-mk-n17.md` of the port project has every command and log):
 
 | boot | wall from `tt-model serve` to READY | notes |
 |---|---:|---|
-| first (cold: empty `/cache` and `/weight-cache`) | 90 s (77 s for the round-0 build of the same code) | `Model built in 16.4 s` (checkpoint → device-layout plan + 8.4 GB `.tensorbin` tier), JIT of every kernel of the four traces + trace capture 66.7 s (52.6 s in round 0), 5 warm-ups; smoke PASS, 50-request bench 69.5 / 84.4 ms device / total, clean stop |
-| second (warm caches, minutes later) | 11–21 s | weight tier and kernels from the persistent caches (`Model built in 1.2 s`, capture 3.4–12.3 s); smoke PASS, 50-request bench 69.6–69.7 / 82.2–82.8 ms device / total, clean stop 1.5–2.6 s |
+| first (cold: empty `/cache` and `/weight-cache`) | 64 s | `Model built in 14.0 s` (checkpoint → device-layout plan + `.tensorbin` tier + the bfp8_b arena), JIT of every kernel of the four traces incl. the megakernel + trace capture 43.2 s, 5 warm-ups; smoke PASS, 50-request bench 55.9 / 69.1 ms device / total, clean stop 2.1 s |
+| second (warm caches, minutes later) | 14 s | weight tier and kernels from the persistent caches (`Model built in 4.8 s`, capture 3.3 s); smoke PASS, 50-request bench 55.9 / 67.7 ms device / total, clean stop 2.5 s |
 
 `push` makes `code/` and `image/` on the Hub exactly the staged trees and replaces `README.md` with the generated card (everything worth
 keeping lives in `card.description` / `card.quickstart`); `tt-model.yaml`, `SERVING.md`, `GPU_COMPARISON.md`, `LICENSE`,
@@ -159,11 +169,21 @@ keeping lives in `card.description` / `card.quickstart`); `tt-model.yaml`, `SERV
 | route | returns |
 |---|---|
 | `GET /health` | `{"status": "ok" \| "starting", "model": "GR00T-N1.7-3B-p150", "device": "blackhole:0"}` — 200 always; `ok` only after warm-up |
-| `GET /info` | model / version / task (40 actions) / hardware / `stage` (`served: stage1-ttnn-traces` + megakernel note) / `policy` (the `TTPolicy` actually used, incl. `vit_fork`) / `weights` (repo, revision, snapshot dir actually loaded) / `source` (port snapshot commit, server version, tt-metal commit, ttnn version) / `inputs` (the full observation contract: camera keys, `video_delta_indices [-15, 0]`, `frame_order`, `vit_grid_thw [1, 16, 28]`, `frame_sizes` acceptance rule, state groups, `L_max`) / `outputs` (the DROID composition rule) / `noise.default_seed` (42) / `limits` / `device_facts` / `warmup_latency_ms` / `warmup_fidelity_vs_golden` / `license` |
+| `GET /info` | model / version / task (40 actions) / hardware / **`stage`** (`served: stage2-megakernel`, `dit_backend`, `mk_arena_dtype` (resolved: `bfp8_b`), `num_command_queues`, `cq1_uploads`, `worker_l1_size` (1395712), `adapter_mem` (`DRAM`), `weights` {529 tensors, 3007.7 MB, `skipped_ttnn_dit` {534, 1031.9 MB}}, `head_timing_s`, a note) / `policy` (the `TTPolicy` actually used, incl. `dit_backend`, `mk_arena_dtype`, `vit_fork`) / `weights` (repo, revision, snapshot dir actually loaded) / `source` (port snapshot commit, server version, tt-metal commit, ttnn version) / `inputs` (the full observation contract: camera keys, `video_delta_indices [-15, 0]`, `frame_order`, `vit_grid_thw [1, 16, 28]`, `frame_sizes` acceptance rule, state groups, `L_max`) / `outputs` (the DROID composition rule) / `noise.default_seed` (42) / `limits` / `device_facts` / `warmup_latency_ms` / `warmup_fidelity_vs_golden` / `license` |
 | `GET /demo` | `{"request": <ready-to-post demo body incl. the four frames and the reference noise>, "expected": <golden actions + gates>}` |
 | `GET /v1/models` | OpenAI-shaped stub (`{"object": "list", "data": [{"id": "nvidia/GR00T-N1.7-3B", ...}]}`) so the tt-model ready card does not 404; this is not a chat API |
 | `POST /predict` | one observation → one 40-step action chunk (below) |
 
+Serving knobs (all read once in the lifespan; the image pins them in `serve.env`):
+
+| variable | image | meaning |
+|---|---|---|
+| `GR00T_DIT_BACKEND` | `megakernel` | `TTPolicy.dit_backend`: `megakernel` (the DiT denoise as one persistent `generic_op`, Stage 2, the port default since 2026-09-18) or `ttnn` (the Stage-1 TTNN op sequence, for A/B; opens the firmware-default L1). Reported in `/info -> stage` |
+| `GR00T_MK_ARENA_DTYPE` | `auto` (→ `bfp8_b`) | the megakernel's DRAM weight arena: `auto` = the port's per-version table (`bfp8_b` for n17), `bf16`, `bfp8_b` |
+| `GR00T_NUM_CQS` | `2` | command queues of the device: `2` = per-call input writes on CQ 1 overlapping the traces (bit-equal to `1`, `tests/tt/test_mk_2cq.py`) |
+| `GR00T_VIT_FORK` | `1` | `1` = the tuned Qwen3-VL tower, `0` = the verbatim tt_dit tower (5× slower, the oracle) |
+| `GR00T_POLICY`, `GR00T_TRACE_LAYOUT` | `mixed_dit`, `per_stage` | dtype policy / trace layout (the validated pair) |
+
 `POST /predict` request (JSON; unknown fields → 422):
 
 | field | type | meaning |
@@ -185,7 +205,7 @@ Response (200):
  "normalized": false, "embodiment": "oxe_droid_relative_eef_relative_joint", "embodiment_id": 24, "version": "n17", "model": "GR00T-N1.7-3B-p150", "layout": "droid",
  "images": {"exterior_image_1_left": {"frames": 2, "received_hw": [180, 320], "vit_grid_thw": [1, 16, 28]}, "wrist_image_left": {"frames": 2, "received_hw": [180, 320], "vit_grid_thw": [1, 16, 28]}},
  "seq_len": 469, "prompt_tokens": 8, "state_dtype": "float32", "noise_source": "seed", "seed": 42,
- "timing_ms": {"decode": 3.6, "encode": 4.3, "device": 69.9, "decode_actions": 0.9, "total": 78.7}}
+ "timing_ms": {"decode": 3.4, "encode": 10.9, "device": 55.9, "decode_actions": 1.1, "total": 71.3}}
 ```
 
 `actions` are physical, un-normalised targets (float32 as decimal JSON), exactly what the reference `get_action` returns for the DROID
@@ -193,41 +213,62 @@ data config: `eef_9d` = the model's RELATIVE end-effector output (per-step q01/q
 XYZ + rot6d → SE(3)) composed onto the request's `eef_9d` (`T_abs = T_state · T_rel`), i.e. an **absolute pose**; `joint_position` = the
 relative joint output added to the request's `joint_position`; `gripper_position` ABSOLUTE (q01/q99). `timing_ms`: `decode` = base64 +
 four PNG decodes + validation (incl. the frame-size rule), `encode` = the port's host preprocessing (cv2 eval chain on four frames,
-Qwen2-VL patching, tokenizer, mRoPE tables), `device` = `predict_normalized` (input writes, `execute_trace` × 4, one blocking readback),
-`decode_actions` = un-normalisation + SE(3) composition, `total` = handler wall time. Errors: **400** (wrong/missing camera or state
-group, one or three frames per camera, frame outside the accepted ratio band or [64, 4096] sides, wrong state width, non-finite state,
-undecodable image, empty/too-long instruction, other embodiment, `seed` + `noise` together, malformed noise), **422** (schema), **503**
-while starting, **500** `Type: text` on a device failure. Handlers are synchronous and serialised on one lock; batch is 1.
+Qwen2-VL patching, tokenizer, mRoPE tables), `device` = `predict_normalized` (input writes, `execute_trace` × 4 — the fourth replay is the
+megakernel `generic_op` —, one blocking readback), `decode_actions` = un-normalisation + SE(3) composition, `total` = handler wall time.
+Errors: **400** (wrong/missing camera or state group, one or three frames per camera, frame outside the accepted ratio band or [64, 4096]
+sides, wrong state width, non-finite state, undecodable image, empty/too-long instruction, other embodiment, `seed` + `noise` together,
+malformed noise), **422** (schema), **503** while starting, **500** `Type: text` on a device failure. Handlers are synchronous and serialised
+on one lock; batch is 1.
 
 ## 4. Caveats
 
-* **Stage-1 path, tuned tower.** TTNN ops in four Metal traces (`vision` 17.5 ms, `llm` 12.4, `adapter` 5.7,
-  `denoise` 30.7 — `bench_e2e` medians on the snapshot, `code/models/experimental/gr00t/benchmarks/results/bench_e2e_n17_snapshot.json`);
-  DiT matmul weights bfp8_b, everything else bf16. The vision tower is the port's `TunedQwen3VLVisionTT` (HiFi2 + fp32-accumulate
-  matmuls, **HiFi3 on blocks 0–5**, HiFi2 SDPA with (448, 448) chunks, `rotary_embedding_hf`, L1 activations, tuned mergers): 17.5 ms
-  traced vs 87 ms for the verbatim tt-nn `Qwen3VlVisionModel` (`GR00T_VIT_FORK=0`, the oracle; its `vit_deepstack_mergers[0]` sits
-  at 0.99965, below the 0.9998 gate — `docs/plan/n17-fidelity.md`). The persistent-megakernel denoise (Stage 2, `tt/megakernel/`,
-  rung K1: 464 GB/s bf16 / 414 GB/s bfp8 weight streaming, `tests/tt/results/mk_k1_summary.md`) is **not** integrated into
-  `Gr00tTT` and not served.
-* **Fidelity margin.** Final actions reproduce the fp32 golden at PCC 0.99996 (all 17 dims × 40 steps) / 0.99995 on the normalised
-  valid slice (boot/smoke gate 0.999); all 58 rows of the canonical tap comparison pass (55 gated taps; tightest
-  `vit_deepstack_mergers[0]` 0.999849 vs 0.9998, 4.9e-5 above the gate — the device floor on that tap with exact matmuls everywhere
-  is ≈ 0.99985, NVIDIA's own bf16 path 0.99995). On six additional DROID goldens five pass every tap; `t2_s120` misses
-  `llm_layer_first` (0.98998 vs 0.991) and `unnormalized_actions.joint_position` (PCC 0.99977 / max|d| 0.096 vs 0.9998 / 0.083), a
-  pre-existing sample-level miss also seen with the verbatim tower (`tests/tt/results/test_e2e_untraced_taps_vs_golden_t2_s120_n17_*.json`).
+* **Stage-2 path: the DiT denoise is one persistent megakernel.** The fourth trace (`denoise`, 16.5 ms = 4 × 4.12 ms per Euler step)
+  replays one `generic_op` that runs the action encoder, the 32 cross/self DiT blocks, the decoder and the Euler update for all four steps:
+  the block weights (bfp8_b, ~1 GB DRAM arena, `tt/megakernel/arena.py`) are streamed once per step by 8 DRAM-bank reader cores and
+  multicast through hub cores to 96 compute cores of the 11×10 grid (`core_map.py`), LayerNorm is distributed, attention is a per-head
+  flash SDPA on the compute cores (`kernels/ops/head_sdpa.hpp`) with the long-key cross blocks (the 448-key image set) split three ways per
+  head (`mk_k4b_summary.md` §0). What this costs: the device runs with 64 KiB less allocatable L1 per core (`worker_l1_size` 1,395,712 B,
+  the kernel-config ring of the megakernel binaries — the compute-core program is ≈ 131 KB, ≈ 4.5 KB under the 136,192 B ring,
+  `mk_k4b_summary.md` §5), the VL adapter's intermediates move to DRAM (`adapter` trace 6.2 vs 5.7 ms), and the DiT's per-step internals
+  (`dit_out[k]`, block taps, AdaLN) are not observable from the served model — the tap comparison covers the backbone / adapter taps and the
+  10 head taps (30 rows) instead of the Stage-1 path's 58. Gain vs the Stage-1 TTNN denoise on the same tree (`mk_default_summary.md` §5):
+  e2e −19 % (64.4 vs 79.7 ms in the same session; 62.8 ms on the snapshot), device −20 % (55.7 vs 69.8 ms), denoise −46 % (16.5 vs
+  30.7 ms); the plan's −30 % e2e gate is not met. `GR00T_DIT_BACKEND=ttnn` serves the Stage-1 path (`stage1-ttnn-traces`).
+* **Tuned tower.** Vision / LLM / adapter are TTNN ops in three Metal traces (`vision` 17.5 ms, `llm` 12.4, `adapter` 6.2 — `bench_e2e` medians on
+  the snapshot, `code/models/experimental/gr00t/benchmarks/results/bench_e2e_n17_20260918-050929.json`). The vision tower is the port's
+  `TunedQwen3VLVisionTT` (HiFi2 + fp32-accumulate matmuls, **HiFi3 on blocks 0–5**, HiFi2 SDPA with (448, 448) chunks, `rotary_embedding_hf`,
+  L1 activations, tuned mergers): 17.5 ms traced vs 87 ms for the verbatim tt-nn `Qwen3VlVisionModel` (`GR00T_VIT_FORK=0`, the oracle; its
+  `vit_deepstack_mergers[0]` sits at 0.99965, below the 0.9998 gate — `docs/plan/n17-fidelity.md`). Unchanged from the Stage-1 package.
+* **Fidelity margin.** Final actions reproduce the fp32 golden at PCC 0.99996 (all 17 dims × 40 steps) / 0.99995 on the normalised valid
+  slice (boot/smoke gate 0.999); all 30 rows of the canonical tap comparison pass (27 gated taps; tightest `vit_deepstack_mergers[0]`
+  0.999849 vs 0.9998, 4.9e-5 above the gate; lowest `vl_self_attention_out` 0.9745 vs 0.95). Megakernel vs the Stage-1 TTNN denoise on the
+  same device: `action_pred_normalized` PCC ≥ 0.999992 on every DROID sample, per-step velocities ≥ 0.99996
+  (`tests/tt/results/test_mk_e2e_backends_agree_*_n17_20260918-0428*.json`). On six additional DROID goldens the actions pass on five:
+  `t2_s120` misses `llm_layer_first` (0.98998 vs 0.991) and `unnormalized_actions.joint_position` (PCC 0.99976 / max|d| 0.107 vs 0.9998 /
+  0.083), a pre-existing sample-level miss also seen with the Stage-1 path and the verbatim tower; `t1_s120` misses one **megakernel-only**
+  per-dim gate, `pred_velocity[k=2]` dim 9 PCC 0.99588 vs 0.996 (full-tensor PCC 0.99983 vs 0.9997 passes; the Stage-1 path passed the per-dim
+  gate; recorded as a known marginal with floor 0.995 in `tests/tt/test_mk_e2e.py`; that sample's actions pass every gate)
+  (`tests/tt/results/test_e2e_untraced_taps_vs_golden_{t1_s120,t2_s120}_n17_20260918-0418*.json`).
 * **One embodiment, one layout, two cameras × two frames, batch 1.** DROID (`oxe_droid_relative_eef_relative_joint`), static layout
   `droid` (16×28 patches per image, `L_max` 51, `S_pad` 512). The other N1.7 layouts in `configs.py` (`g1_4x3`, `g1_synthetic_16x9`,
   `cam3_4x3`) have no traces in this image and no device validation. A client without history must send its oldest available frame
   twice (the reference wraps `iloc[-15]` at episode start, so results at t < 15 differ from NVIDIA's numbers either way).
 * **Deterministic noise.** Actions are a deterministic function of (observation, seed | noise); two identical requests return
-  bit-identical chunks. The default seed 42 does **not** reproduce the reference `initial_noise` (CUDA draw); `GET /demo` and the smoke
-  test send the reference noise explicitly (`demo/n17/noise.json`, `seed_equivalent: null`).
+  bit-identical chunks; the 2-CQ device produces the same bits as a 1-CQ device on every probe (`test_mk_2cq_bit_equality_n17_20260918-041603.json`,
+  26 rows). The default seed 42 does **not** reproduce the reference `initial_noise` (CUDA draw); `GET /demo` and the smoke test send the
+  reference noise explicitly (`demo/n17/noise.json`, `seed_equivalent: null`).
 * **Host cost.** N1.7's `encode` (four 256×455 frames through the cv2 chain, torchvision bicubic patching, tokenizer, cached mRoPE / vision
-  tables) is 4.9 ms median / 11.2 p90 in the port benchmark and 8–10 ms median / 20–33 ms p90 inside the serving process, vs 1.5–3 ms
-  for the GR1 packages; `decode` of four PNGs ≈ 3.5 ms (6 ms on the first request). Both run outside the device lock. Inside the
-  serving process torch's 16 threads compete with the ttnn worker threads (`GR00T_TORCH_THREADS` is the knob; unset in this image).
-* **First boot is cold**: the checkpoint is converted to the device layout into `/weight-cache` (~8.4 GB) and every kernel is JIT-compiled
-  into `/cache`; both persist under `~/.cache/tt-model/gr00t-n1.7-3b-p150/`. Measured boot times: §2.
+  tables) is 5.6 ms median / 11.6 p90 in the port benchmark on an idle host (8.5 / 19.2 in a run taken while a CPU suite was finishing) and
+  8.7 ms median / 12.1 ms p90 inside the serving process, vs 1.5–2 ms for the GR1 packages; `decode` of four
+  PNGs ≈ 3.4 ms. Both run outside the device lock. Inside the serving process torch's threads compete with the ttnn worker
+  threads (`GR00T_TORCH_THREADS` is the knob; unset in this image). With the device time down to ~56 ms the host share of a served call is
+  now about a fifth.
+* **Megakernel-side open items** (kernel work, not changed by this package): the compute-core binaries are ≈ 4.5 KB under the kernel-config
+  ring, so any further kernel growth needs a size check (`mk_k4b_summary.md` §5); building a second megakernel model in one process on the
+  same open device after releasing the first hung the card once on N1.6 during development (`mk_default_summary.md` §10) — the server builds
+  exactly one model per process, as every shipped path does.
+* **First boot is cold**: the checkpoint is converted to the device layout into `/weight-cache` (2.9 GB) and every kernel incl. the megakernel
+  is JIT-compiled into `/cache` (483 MB); both persist under `~/.cache/tt-model/gr00t-n1.7-3b-p150/`. Measured boot times: §2.
 * **Python 3.12 image vs 3.10 host venv.** The image resolves the packages on 3.12 (`requirements.lock`); the host venv is 3.10. The
   code is 3.10/3.12 compatible; the image is the authoritative validation (§2 numbers).
 * **Vendored tokenizer / processor files** (`code/gr00t_p150/assets/tokenizer/n17`, Apache-2.0 — see `PROVENANCE.md` there and
@@ -236,11 +277,14 @@ while starting, **500** `Type: text` on a device failure. Handlers are synchrono
 
 ## 5. Where the numbers come from
 
-Everything in the card is measured and shipped: `code/models/experimental/gr00t/benchmarks/results/bench_e2e_n17_snapshot.json`
-(Stage-1 latency on the snapshot, 50 calls, tt-metal `668c2907575`), `code/models/experimental/gr00t/tests/tt/results/test_e2e_{predict_actions_vs_golden,
-traced_equals_untraced,untraced_taps_vs_golden}_n17_20260917-2053*.json` (canonical fidelity, traced == untraced), the six
-`test_e2e_untraced_taps_vs_golden_<sample>_n17_20260917-2055*.json` / `-2056*` / `-2057*` / `-2058*` / `-2059*` / `-2100*` files (multi-sample), `test_stale_state_*_n17_20260917-2054*.json`,
-`test_determinism_traced_and_untraced_n17_20260917-2055*.json`, `tests/tt/results/{d2_summary,p_summary,mk_k1_summary}.md`,
+Everything in the card is measured and shipped under `code/models/experimental/gr00t/`: `benchmarks/results/bench_e2e_n17_20260918-050929.json`
+(Stage-2 latency on the snapshot, 50 calls, idle host; `…-050724.json` the first run with a busier host), `benchmarks/results/bench_e2e_n17_20260918-032335.json`
+(the Stage-1 TTNN path on the same tree, same day) and `…-043104.json` (the megakernel on the live tree in the same session),
+`tests/tt/results/test_e2e_{predict_actions_vs_golden,traced_equals_untraced,untraced_taps_vs_golden}_n17_20260918-0452*.json` (canonical fidelity on
+the snapshot commit `009de892c895`, traced == untraced), the six `test_e2e_untraced_taps_vs_golden_<sample>_n17_20260918-0418*.json` files (multi-sample,
+K4b-checkpoint tree = the snapshot's code), `test_mk_e2e_backends_agree_*_n17_20260918-0428*.json` (megakernel vs TTNN denoise),
+`test_mk_2cq_bit_equality_n17_20260918-041603.json`, `test_stale_state_*_n17_20260918-0420*.json`, `test_determinism_traced_and_untraced_n17_20260918-042138.json`,
+`tests/tt/results/{mk_default_summary,mk_k4b_summary,mk_k5_summary,d2_summary,p_summary}.md` (the megakernel's design, measurements and decisions),
 `code/gr00t_p150/demo/n17/expected.json` (fp32 golden + official bf16 reference actions for the demo step). The RTX 5090 rows are in
 `GPU_COMPARISON.md`. The served numbers of this image (`timing_ms` over 50 requests, cold / warm boot) were measured on the build host as
-described in §2 and recorded in `docs/publish/build-n17.md` of the port project.
+described in §2 and recorded in `docs/publish/build-mk-n17.md` of the port project.

tt-model.yaml

@@ -1,6 +1,8 @@
 # SPDX-License-Identifier: Apache-2.0
 # tt-model-manager container manifest (schema 5.1) for NVIDIA Isaac GR00T N1.7-3B (DROID layout) on Blackhole
-# (Stage-1 TTNN trace path, tuned Qwen3-VL vision tower).
+# (Stage-2 release: the whole DiT denoise runs as ONE persistent megakernel generic_op on a bfp8_b DRAM weight arena;
+# vision / LLM / adapter are TTNN ops in Metal traces; tuned Qwen3-VL vision tower). The Stage-1 TTNN denoise stays
+# selectable with GR00T_DIT_BACKEND=ttnn.
 #
 # Run EVERY tt-model command from this directory: `source.tt_metal` and `extra_code[].root: code` resolve against the
 # process CWD, not against this file. `--out` must lie outside any git checkout (`package` deletes <out>/<name> first).
@@ -8,11 +10,11 @@
 #
 #   tt-model package --container tt-model.yaml --out /home/deepgadget/experiments/gr00t/publish/build   # minutes with the ccache mount, ~2 h cold
 #   tt-model serve   /home/deepgadget/experiments/gr00t/publish/build/gr00t-n1.7-3b-p150/tt_kernel_manifest.json
-#   python code/gr00t_p150/server/smoke_test.py --url http://127.0.0.1:20000      # PASS/FAIL one-liner
+#   python code/gr00t_p150/server/smoke_test.py --url http://127.0.0.1:20000      # PASS/FAIL one-liner (prints /info -> stage)
 #   tt-model stop    /home/deepgadget/experiments/gr00t/publish/build/gr00t-n1.7-3b-p150/tt_kernel_manifest.json   # org/name works only after pull/push
 #   tt-model push    /home/deepgadget/experiments/gr00t/publish/build/gr00t-n1.7-3b-p150 --publish
 #
-# The one Blackhole card of the build host is shared: `serve -> smoke -> stop` runs inside ONE bin/with-device.sh hold.
+# The one Blackhole card of the build host is shared: `serve -> smoke -> bench -> stop` runs inside ONE bin/with-device.sh hold.
 schema: "5.1"
 
 repo: changh95/GR00T-N1.7-3B-p150
@@ -40,7 +42,8 @@ source:
   # Clean shared clone of tt-metal main 668c2907575 (v0.79.0-dev20260914-1), the commit the port was validated on;
   # git status is empty there, so the card records a clean tree. The image builds tt-metal from this source.
   tt_metal: /home/deepgadget/experiments/gr00t/publish/tt-metal-668c2907575
-  # tt-metal-relative files the port imports from the tree OTHER than itself, with their import closure.
+  # tt-metal-relative files the port imports from the tree OTHER than itself, with their import closure
+  # (grep of every `from models.` / `import models.` outside models.experimental.gr00t in the snapshot).
   code:
     - models/common/utility_functions.py                        # models.tt_dit.utils.matmul imports is_blackhole
     - models/tt_dit/utils/matmul.py                             # tt/layers.py: MinimalMatmul config rows (lazy)
@@ -57,12 +60,19 @@ source:
     - models/tt_dit/parallel/config.py
     - models/tt_dit/parallel/manager.py
     - models/tt_dit/encoders/qwen3vl/vision_qwen3vl.py          # tt/qwen3vl_vision.py: verbatim tower (lazy; the served default is the tuned fork in the port itself)
-    - models/demos/deepseek_v3_b1/unified_kernel_descriptor.py  # tt/megakernel/descriptors.py (lazy; Stage-2 work, not served)
+    - models/demos/deepseek_v3_b1/unified_kernel_descriptor.py  # tt/megakernel/descriptors.py: the megakernel's UnifiedKernelDescriptor (SERVED: builds the denoise generic_op)
+    # The megakernel's device kernel (tt/megakernel/kernels/dit_kernel.cpp) #includes these four tree headers by relative path
+    # (../../../../../demos/deepseek_v3_b1/unified_kernels/...); the JIT compiles them at boot, so they must be in the image
+    # (round 0 of the megakernel package failed its first boot on exactly this: docs/publish/build-mk-n17.md).
+    - models/demos/deepseek_v3_b1/unified_kernels/kernel_op_api.hpp
+    - models/demos/deepseek_v3_b1/unified_kernels/kernel_utils.hpp
+    - models/demos/deepseek_v3_b1/unified_kernels/dataflow_utils.hpp
+    - models/demos/deepseek_v3_b1/unified_kernels/mcast.hpp
   # This repo's own code (lands at /opt/tt-metal/<path>; PYTHONPATH=/opt/tt-metal).
   extra_code:
     - root: code
       paths:
-        - models        # code/models/experimental/gr00t: the frozen snapshot of the port (tt-metal commit 5833d4be, parent 668c2907575)
+        - models        # code/models/experimental/gr00t: the frozen snapshot of the port (tt-metal ref gr00t-p150-snapshot-mk = 009de892c895, parent 668c2907575; K4b kernel tree 7df61c35e9dc7857)
         - gr00t_p150    # FastAPI server, schemas, smoke test, DROID demo observation, vendored Qwen2 tokenizer + Qwen2-VL processor config (n17)
         - scripts       # download_weights.sh, bench_http.py
   ubuntu: "22.04"
@@ -95,14 +105,17 @@ serve:
     GR00T_VERSION: "n17"                   # the server code is shared with the N1.5 / N1.6 packages and never guesses
     GR00T_EMBODIMENT: "oxe_droid_relative_eef_relative_joint"   # DROID (Franka arm, relative EEF + relative joints), embodiment slot 24; the only device-validated N1.7 embodiment
     GR00T_LAYOUT: "droid"                  # static layout: 2 cameras x 2 frames -> 4 images x 112 tokens (16x28 patches each), L_max 51 text tokens, S_pad 512
-    GR00T_POLICY: "mixed_dit"              # TTPolicy.dtype_policy: DiT matmul weights bfp8_b, everything else bf16 (validated Stage-1 default)
-    GR00T_TRACE_LAYOUT: "per_stage"        # four Metal traces: vision / llm / adapter / denoise (the benchmarked layout)
+    GR00T_POLICY: "mixed_dit"              # TTPolicy.dtype_policy: DiT matmul weights bfp8_b, everything else bf16 (the validated default)
+    GR00T_TRACE_LAYOUT: "per_stage"        # four Metal traces: vision / llm / adapter / denoise (the denoise trace holds the megakernel generic_op)
+    GR00T_DIT_BACKEND: "megakernel"        # TTPolicy.dit_backend (the port default since 2026-09-18): the whole DiT denoise as ONE persistent generic_op; "ttnn" = the Stage-1 TTNN op sequence (A/B, tests/tt/results/mk_default_summary.md)
+    GR00T_MK_ARENA_DTYPE: "auto"           # the megakernel's DRAM weight arena: auto -> the port's per-version table = bfp8_b for n17 (the validated configuration; bf16 selectable); /info -> stage.mk_arena_dtype reports the resolved value
+    GR00T_NUM_CQS: "2"                     # command queues of the device (tt.model.open_model_device): per-call input writes on CQ 1 overlap the traces; bit-equal to 1 CQ (tests/tt/test_mk_2cq.py)
     GR00T_VIT_FORK: "1"                    # tuned Qwen3-VL tower (HiFi2 + fp32 acc, HiFi3 on blocks 0-5; the port default since 2026-09-17); "0" = the verbatim tt_dit tower (oracle, 5x slower)
     GR00T_TOKENIZER_DIR: "/opt/tt-metal/gr00t_p150/assets/tokenizer/n17"   # vendored Qwen2 tokenizer + Qwen2-VL preprocessor_config.json (not in the weights repo)
     GR00T_PROJECT_ROOT: "/nonexistent"     # kill the dev-box default the port's prompts.py would otherwise fall back to
     GR00T_GOLDEN_ROOT: "/nonexistent"      # same for common/golden.py (tests only; defensive)
-    GR00T_TT_CACHE: "/weight-cache/gr00t-tt"   # host .pt plan tier + .tensorbin device tier (~8.4 GB) -> ~/.cache/tt-model/gr00t-n1.7-3b-p150/weights
-    GR00T_WARMUP_RUNS: "5"                 # traced predict() calls after trace capture, before READY (server-n17.md)
+    GR00T_TT_CACHE: "/weight-cache/gr00t-tt"   # host .pt plan tier + .tensorbin device tier (~2.9 GB under the megakernel: the ttnn DiT tensors are not converted; 8.4 GB on the Stage-1 path) -> ~/.cache/tt-model/gr00t-n1.7-3b-p150/weights
+    GR00T_WARMUP_RUNS: "5"                 # traced predict() calls after trace capture, before READY
     GR00T_WARMUP_PCC_MIN: "0.999"          # boot FAILS when the warm-up actions' PCC vs the shipped fp32 golden is below this
 
 # Build-time assertions, run INSIDE the finished image as uid 1000, no device, no weights, no HF_MODEL.
@@ -113,6 +126,14 @@ verify:
   - "from models.experimental.gr00t.common import configs, preprocessing, normalization, prompts, checkpoint, weights; assert configs.HF_SNAPSHOT_SHAS['n17'] == '2fc962b973bccdd5d8ce4f67cc63b264d6886495' and configs.HF_REPOS['n17'] == 'nvidia/GR00T-N1.7-3B'"
   - "from models.experimental.gr00t.common.configs import get_config; c = get_config('n17'); l = c.canonical_layout; assert (l.name, c.canonical_embodiment, l.embodiment_id, l.n_cameras, l.n_images, l.grid_thw, l.tokens_per_image, l.l_max, l.s_pad_llm, l.noise_shape, l.valid_slice) == ('droid', 'oxe_droid_relative_eef_relative_joint', 24, 2, 4, (1, 16, 28), 112, 51, 512, (1, 40, 132), (40, 17)), l"
   - "from models.experimental.gr00t.tt.qwen3vl_vision import build_vision_tower, resolve_vit_fork, TunedVisionConfig; from models.experimental.gr00t.tt.policy import TTPolicy; assert TTPolicy().vit_fork is True and resolve_vit_fork(TTPolicy(), None, 'qwen3vl') is True"
+  - "from models.experimental.gr00t.tt.policy import TTPolicy, DEFAULT_DIT_BACKEND, MK_ARENA_DTYPE_DEFAULT; p = TTPolicy(); assert DEFAULT_DIT_BACKEND == 'megakernel' and p.dit_backend == 'megakernel' and p.mk_arena_dtype == 'auto' and p.mk_arena_dtype_for('n17') == 'bfp8_b' == MK_ARENA_DTYPE_DEFAULT['n17'], p.to_dict()"
+  - "from models.experimental.gr00t.tt.policy import TTPolicy; assert TTPolicy().with_env_overrides({'GR00T_DIT_BACKEND': 'ttnn'}).dit_backend == 'ttnn' and TTPolicy().with_env_overrides({'GR00T_MK_ARENA_DTYPE': 'bf16'}).mk_arena_dtype == 'bf16'"
+  - "import models.experimental.gr00t.tt.megakernel.dit_program as dp; from models.experimental.gr00t.tt.megakernel import arena, core_map, descriptors; assert dp.DiTMegakernel and descriptors.L1_USABLE_BYTES == 1461248 and core_map.N_COMPUTE_DEFAULT == 96"
+  - "from models.experimental.gr00t.tt.model import open_model_device, worker_l1_size_for, MK_L1_CUT_BYTES_DEFAULT, ADAPTER_MEM_BY_BACKEND; from models.experimental.gr00t.tt.policy import TTPolicy; assert MK_L1_CUT_BYTES_DEFAULT == 65536 and worker_l1_size_for(TTPolicy()) == 1395712 and worker_l1_size_for(TTPolicy(dit_backend='ttnn')) is None and ADAPTER_MEM_BY_BACKEND == {'ttnn': 'L1', 'megakernel': 'DRAM'}"
+  - "import hashlib; from pathlib import Path; d = Path('/opt/tt-metal/models/experimental/gr00t'); rows = [l.split() for l in (d / 'tests/tt/results/mk_tree_20260918-041439.sha256').read_text().splitlines() if l.strip()]; assert len(rows) == 16 and all(hashlib.sha256((d / p).read_bytes()).hexdigest() == h for h, p in rows), 'tt/megakernel tree != the K4b checkpoint 7df61c35e9dc7857'"
+  - "import os, re; from pathlib import Path; k = Path('/opt/tt-metal/models/experimental/gr00t/tt/megakernel/kernels/dit_kernel.cpp'); incs = re.findall(r'#include \"(\\.\\./[^\"]+)\"', k.read_text()); assert len(incs) == 4 and all(Path(os.path.normpath(k.parent / i)).is_file() for i in incs), incs"
+  - "import inspect, gr00t_p150.server.app as a; src = inspect.getsource(a); assert a.STAGE_SERVED == {'megakernel': 'stage2-megakernel', 'ttnn': 'stage1-ttnn-traces'} and a.DEFAULT_NUM_CQS == 2 and 'device = open_model_device(' in src and 'GR00T_DIT_BACKEND' in src and 'GR00T_NUM_CQS' in src"
+  - "import os; os.environ.update({'GR00T_VERSION': 'n17', 'TT_WEIGHTS_REVISION': '2fc962b973bccdd5d8ce4f67cc63b264d6886495', 'GR00T_DIT_BACKEND': 'megakernel', 'GR00T_MK_ARENA_DTYPE': 'auto', 'GR00T_NUM_CQS': '2', 'GR00T_VIT_FORK': '1'}); from gr00t_p150.server.app import load_config; c = load_config(); assert (c.version, c.dit_backend, c.mk_arena_dtype, c.num_cqs, c.vit_fork) == ('n17', 'megakernel', 'auto', 2, True), c"
   - "import models.tt_dit.utils.matmul as mm; assert mm.get_matmul_config"
   - "from models.tt_dit.encoders.qwen3vl.vision_qwen3vl import Qwen3VlVisionModel; assert Qwen3VlVisionModel"
   - "import models.demos.deepseek_v3_b1.unified_kernel_descriptor as u; assert u.UnifiedKernelDescriptor"
@@ -126,12 +147,12 @@ verify:
   - "from gr00t_p150.server.app import load_demo_request; import numpy as np; r = load_demo_request('n17'); assert set(r['images']) == {'exterior_image_1_left', 'wrist_image_left'} and all(len(v) == 2 for v in r['images'].values()) and np.asarray(r['noise']).squeeze().shape == (40, 132)"
   - "from pathlib import Path; assert Path('/opt/tt-metal/models/experimental/gr00t/common/golden_tap_map.json').is_file() and Path('/opt/tt-metal/models/experimental/gr00t/tests/tt/gates/gates_multi.json').is_file()"
   - "from pathlib import Path; assert Path('/opt/tt-metal/gr00t_p150/server/smoke_test.py').is_file() and Path('/opt/tt-metal/scripts/bench_http.py').is_file() and Path('/opt/tt-metal/scripts/download_weights.sh').is_file()"
-  - "import ttnn; assert all(hasattr(ttnn, a) for a in ('begin_trace_capture', 'end_trace_capture', 'execute_trace', 'release_trace', 'copy_host_to_device_tensor', 'MinimalMatmulConfig', 'embedding'))"
+  - "import ttnn; assert all(hasattr(ttnn, a) for a in ('begin_trace_capture', 'end_trace_capture', 'execute_trace', 'release_trace', 'copy_host_to_device_tensor', 'MinimalMatmulConfig', 'embedding', 'generic_op', 'allocate_tensor_on_device'))"
   - "from gr00t_p150.server.app import parse_mesh_shape as p; assert p('1x1') == p('(1, 1)') == p('1,1') == (1, 1)"
 
 card:
   description: |
-    NVIDIA Isaac GR00T N1.7 (3B vision-language-action policy) running on one Tenstorrent Blackhole p150a via tt-nn, in the DROID layout: two cameras × two frames (t−15 and t) at 16:9, a 17-dim Franka state and a task instruction in, a 40-step chunk of end-effector, gripper and joint actions out. Stage-1 release: TTNN ops replayed from four Metal traces (DiT weights bfp8, everything else bf16) with a tuned Qwen3-VL tower; the persistent-megakernel denoise is in progress and not in this image.
+    NVIDIA Isaac GR00T N1.7 (3B vision-language-action policy) running on one Tenstorrent Blackhole p150a via tt-nn, in the DROID layout: two cameras × two frames (t−15 and t) at 16:9, a 17-dim Franka state and a task instruction in, a 40-step chunk of end-effector, gripper and joint actions out. Stage-2 release: the whole DiT denoise runs as one persistent megakernel streaming bfp8 weights from a DRAM arena; vision (tuned Qwen3-VL tower), LLM and adapter are TTNN ops in Metal traces; `GR00T_DIT_BACKEND=ttnn` selects the Stage-1 denoise.
     Weights: [nvidia/GR00T-N1.7-3B](https://huggingface.co/nvidia/GR00T-N1.7-3B) · Paper: [arXiv:2503.14734](https://arxiv.org/abs/2503.14734) · Upstream code: [NVIDIA/Isaac-GR00T](https://github.com/NVIDIA/Isaac-GR00T) (`main`, the `n1.7-release` line) · Port: `code/models/experimental/gr00t` and the server `code/gr00t_p150` in this repo
   quickstart: |
     ### Run with tt-cli
@@ -151,16 +172,16 @@ card:
     ```
 
     - `POST /predict`: `images` (`{"exterior_image_1_left": [<base64 PNG/JPEG at t−15>, <at t>], "wrist_image_left": [...]}`, exactly two frames per camera, 16:9 frames), `state` (`{group: [floats]}`: `eef_9d` 9 = xyz + rot6d, `gripper_position` 1, `joint_position` 7, raw values), `instruction` (≤ 51 tokens); optional `seed` (default 42) or `noise` (`[40, 132]`), `return_normalized`.
-    - `GET /health`, `GET /info`, `GET /demo` (the shipped demo request with its reference noise, and its fp32 reference actions).
+    - `GET /health`, `GET /info` (`stage` says what is served), `GET /demo` (the shipped demo request with its reference noise, and its fp32 reference actions).
 
     ### Response
 
     ```json
-    {"actions": {"eef_9d":           [[0.5284, 0.0182, 0.4569, -0.0784, 0.3928, -0.9163, 0.9923, 0.1188, -0.0340], "... 40 steps"],
-                 "gripper_position": [[0.0], [0.0039], "..."],
-                 "joint_position":   [[-0.0044, 0.0347, 0.0008, -1.9162, 0.1634, 1.6095, 0.0847], "..."]},
+    {"actions": {"eef_9d":           [[0.5284, 0.0181, 0.4569, -0.0786, 0.3927, -0.9163, 0.9923, 0.1193, -0.034], "... 40 steps"],
+                 "gripper_position": [[0.0], [0.002], "..."],
+                 "joint_position":   [[-0.0051, 0.034, 0.0013, -1.917, 0.1641, 1.6101, 0.0844], "..."]},
      "action_horizon": 40, "embodiment": "oxe_droid_relative_eef_relative_joint", "seq_len": 469, "noise_source": "client", "seed": null,
-     "timing_ms": {"decode": 3.6, "encode": 4.3, "device": 69.9, "decode_actions": 0.9, "total": 78.7}}
+     "timing_ms": {"decode": 3.4, "encode": 10.9, "device": 55.9, "decode_actions": 1.1, "total": 71.3}}
     ```
 
     - The example is the demo request with the reference noise, as `GET /demo` returns it (`noise_source: "client"`, `seed: null`); `timing_ms` from a warm request of the same serve.
@@ -176,20 +197,22 @@ card:
 
     | Metric | Value |
     |---|---:|
-    | Action chunk vs the fp32 reference (same observation, same initial noise) | PCC `eef_9d` 0.99998 · `joint_position` 0.99996 · `gripper_position` max\|d\| 0.0038 (near-constant group) · all 17 dims × 40 steps 0.99996 · normalised valid slice 0.99995 |
-    | Intermediate taps vs the fp32 reference (55 gated taps, from pixels) | 58 of 58 rows pass; tightest `vit_deepstack_mergers[0]` 0.999849 vs gate 0.9998. On six more DROID goldens five pass every tap; `t2_s120` misses two marginal gates (`llm_layer_first` 0.98998 vs 0.991, `joint_position` PCC 0.99977 / max\|d\| 0.096 vs 0.9998 / 0.083), pre-existing |
-    | Traced vs untraced, determinism, stale state | bit-identical; 5 trace replays bit-identical; A→B→A→B observation switches reproduce A and B exactly |
-    | Inference on p150a (port benchmark, warm, batch 1, median of 50) | **75.8 ms end-to-end** (p90 81.6) ≈ host encode 4.9 + upload 3.2 + traces 66.1 (vision 17.5 · LLM 12.4 · adapter 5.7 · denoise 30.7) + readback and decode 1.2 |
-    | Served over HTTP by this image (warm, batch 1, 50 requests) | 69.5 ms device (p90 69.7) · 84.4 ms server-side incl. four PNG decodes (p90 94.5) |
-    | Same policy on an RTX 5090 (same host, NVIDIA's unmodified N1.7 policy, eager PyTorch bf16, idle machine) | `get_action` 59.8–65.6 ms model, 73–79 ms incl. host pre/post-processing → p150a 1.01× faster end-to-end; NVIDIA's published H100 TensorRT full pipeline: 27.9 ms (one camera) |
+    | Action chunk vs the fp32 reference (same observation, same initial noise) | PCC `eef_9d` 0.99998 · `joint_position` 0.99996 · `gripper_position` max\|d\| 0.0031 (near-constant group) · all 17 dims × 40 steps 0.99996 · normalised valid slice 0.99995 |
+    | Intermediate taps vs the fp32 reference (27 gated taps, from pixels; DiT internals are not observable inside the megakernel) | 30 of 30 rows pass; tightest `vit_deepstack_mergers[0]` 0.999849 vs gate 0.9998. On six more DROID goldens the actions pass on five: `t2_s120` has two pre-existing misses (`llm_layer_first` 0.98998 vs 0.991, `joint_position` PCC 0.99976 / max\|d\| 0.107 vs 0.9998 / 0.083); `t1_s120` adds one megakernel-only marginal tap, `pred_velocity[k=2]` dim 9 per-dim PCC 0.99588 vs 0.996 (its actions pass) |
+    | Megakernel vs Stage-1 TTNN denoise (same device, 7 DROID samples) | `action_pred_normalized` PCC ≥ 0.999992 between the two backends; per-step velocities ≥ 0.99996 |
+    | Traced vs untraced, determinism, stale state, 2 vs 1 command queue | bit-identical; 5 trace replays bit-identical; A→B→A→B reproduces A and B exactly; 2-CQ == 1-CQ on every probe (26 rows) |
+    | Inference on p150a (port benchmark, warm, batch 1, median of 50) | **62.8 ms end-to-end** (p90 68.6) ≈ host encode 5.6 + upload 3.2 + traces 52.4 (vision 17.5 · LLM 12.4 · adapter 6.2 · denoise 16.5 = 4 × 4.12) + readback and decode 1.3. Stage-1 path on the same tree: 79.7 ms (denoise 30.7) |
+    | Served over HTTP by this image (warm, batch 1, 50 requests) | 55.9 ms device (p90 56.3) · 69.1 ms server-side incl. four PNG decodes (p90 72.7) |
+    | Same policy on an RTX 5090 (same host, NVIDIA's unmodified N1.7 policy, eager PyTorch bf16, idle machine) | `get_action` 59.8–65.6 ms model, 73–79 ms incl. host pre/post-processing → p150a 1.22× faster end-to-end; NVIDIA's published H100 TensorRT full pipeline: 27.9 ms (one camera) |
 
     ### Caveats
 
-    - Stage-1 path: TTNN ops in four Metal traces, DiT matmul weights bfp8, everything else bf16. The vision tower is the port's tuned Qwen3-VL fork (HiFi3 matmuls on blocks 0–5; 17.5 ms vs 87 ms for the verbatim tower, `GR00T_VIT_FORK=0` selects it). The megakernel denoise (weight streaming measured at 464 GB/s bf16 / 414 GB/s bfp8 over the 32-block weight set) is not in this image.
+    - Stage-2 path: one persistent `generic_op` runs the whole denoise inside the fourth Metal trace, streaming the 32 blocks' bfp8 weights once per step from a ~1 GB DRAM arena over the 11×10 Tensix grid (96 compute cores, 8 DRAM readers, hubs), with distributed LayerNorm, per-head flash SDPA and the cross attention over the 448-key image set split three ways. Costs: 64 KiB less allocatable L1 per core (`worker_l1_size` 1,395,712), the VL adapter's intermediates in DRAM (+0.6 ms), compute binaries of ~131 KB sitting ~4.5 KB under the 136 KB kernel-config ring, and no DiT-intermediate taps. Gain vs the Stage-1 path on the same tree: −19 % end-to-end, −46 % denoise (the plan's −30 % e2e was not reached). Everything outside the DiT matmuls is bf16.
+    - The vision tower is the port's tuned Qwen3-VL fork (HiFi3 matmuls on blocks 0–5; 17.5 ms vs 87 ms for the verbatim tower, `GR00T_VIT_FORK=0` selects it).
     - One embodiment and layout: DROID (`oxe_droid_relative_eef_relative_joint`, embodiment slot 24), exactly two cameras × two frames (t−15 and t), landscape frames that land on the 16×28-patch grid (width / height in [1.686, 1.814]: 16:9, e.g. 320×180), instruction ≤ 51 tokens; anything else is refused with 400. Batch 1.
-    - Noise: the default `seed` is 42 (CPU generator, deterministic). The reference initial noise shipped with the demo (`code/gr00t_p150/demo/n17/noise.json`) was drawn on a CUDA generator and is not seed-reproducible; send it as `noise` to reproduce the reference actions.
+    - Noise: default `seed` 42 (CPU generator, deterministic). The demo's reference noise (`code/gr00t_p150/demo/n17/noise.json`) was drawn on CUDA and is not seed-reproducible; send it as `noise` to reproduce the reference actions.
     - Not an OpenAI-compatible API; `GET /v1/models` is a stub.
-    - First boot is cold (weight conversion + kernel JIT, 90 s); later boots take 11–21 s. Validated on tt-metal `668c2907575`, single p150a only. Details: [`SERVING.md`](SERVING.md), [`GPU_COMPARISON.md`](GPU_COMPARISON.md).
+    - First boot is cold (weight conversion + kernel JIT, 64 s); later boots take 14 s. Validated on tt-metal `668c2907575` (port snapshot `009de892c895`), single p150a only. Details: [`SERVING.md`](SERVING.md), [`GPU_COMPARISON.md`](GPU_COMPARISON.md).
 
     ### Licensing