Stage-2 release: DiT denoise as one persistent megakernel (snapshot 009de892, image 0be2c39e4a01)

Replaces the Stage-1 (TTNN-op denoise) package with the megakernel release: image sha256:0be2c39e4a01f307b061822a3a6d0b098e495a20c8ca0870953f0a73a05e4e60 (tt-model/gr00t-n1.6-3b-p150:0be2c39e4a01), port snapshot 009de892c895 on tt-metal 668c290757, code/ digest 8e6bd13a23654f91. `tt-model push --publish` uploaded code/, image/, tt_kernel_manifest.json, requirements.lock and the generated card and pruned the code/ and image/ files this bundle no longer ships; this commit restores the reviewed staging card (front matter: license other / nvidia-license, pipeline_tag robotics, base_model nvidia/GR00T-N1.6-3B, 13 tags; body = the tt-model.yaml card with the simplified quickstart bullets) and the authored files tt-model.yaml, SERVING.md, GPU_COMPARISON.md, LICENSE, LICENSE-NOTICE.md, requirements.lock and media/*. Card text per docs/publish/review-mk.md.

GPU_COMPARISON.md

@@ -1,8 +1,9 @@
 # GR00T-N1.6-3B-p150 — Blackhole p150a vs RTX 5090 (same host, same weights, same observation)
 
 Facts only. The GPU rows were measured on the build host during the port's reference-environment study
-(2026-09-14, `docs/research/ref-env-n16.md` §5 of the port project); the p150a rows are the port's own Stage-1
-benchmark and the served numbers of this image. Nothing here was re-measured for this file; every number names its source.
+(2026-09-14, `docs/research/ref-env-n16.md` §5 of the port project); the p150a rows are the port's own benchmark of the
+frozen snapshot this image ships (Stage-2: megakernel denoise), the same-tree TTNN-denoise run it replaced, and the served
+numbers of this image. The GPU numbers were not re-measured for this file; every number names its source.
 
 ## What was run
 
@@ -15,7 +16,8 @@ benchmark and the served numbers of this image. Nothing here was re-measured for
 | GPU software | Python 3.10.20, torch 2.7.1+cu128 (cuDNN 90701), transformers 4.51.3, flash_attn 2.7.4.post1 (sm_120 SASS, imports and runs natively), numpy 1.26.4 — the `uv.lock` of `Isaac-GR00T-n1.6` (`ref-env-n16.md` §1) |
 | GPU loop | medians of **20 warm iterations** after warm-up, `torch.cuda.synchronize()` around each call, one process per configuration |
 | GPU timing definitions | `e2e` = `policy.get_action` incl. ~2.5 ms of CPU preprocessing (albumentations, PIL resize, tokenisation) and the numpy decode; `backbone` and `action head` are the GPU sub-steps (`action head` = 4 DiT steps + encoders/decoder) |
-| p150a (device) | one Blackhole p150a, tt-metal `668c2907575`, the port's `benchmarks/bench_e2e.py` (Stage-1: TTNN ops in four Metal traces, `mixed_dit` = bfp8_b DiT matmul weights / bf16 elsewhere, `per_stage` layout): 10 warm-up calls, then **50 calls** of `encode → upload (+sync) → 4 × execute_trace → one sync → D2H → decode`, host wall clock medians; a second 50-call pass with a sync after every trace gives the per-trace split — `code/models/experimental/gr00t/benchmarks/results/e2e_stage1_n16.json` |
+| p150a (device, Stage 2 = this image's path) | one Blackhole p150a, the frozen snapshot `009de892c895` of the port on tt-metal `668c2907575`, `benchmarks/bench_e2e.py` defaults: `dit_backend=megakernel` (the DiT denoise as one persistent `generic_op`, bfp8_b DRAM weight arena), `mixed_dit` elsewhere (bf16), `per_stage` traces, 2 command queues with CQ-1 input uploads, device embedding gather, `worker_l1_size` 1,395,712: 10 warm-up calls, then **50 calls** of `encode → upload → 4 × execute_trace → D2H → decode`, host wall clock medians; a second 50-call pass with a sync after every trace gives the per-trace split — `code/models/experimental/gr00t/benchmarks/results/bench_e2e_n16_20260918-050632.json` |
+| p150a (device, TTNN denoise on the same tree) | the same tree and settings with `--dit-backend ttnn` (the Stage-1 op sequence in the `denoise` trace, firmware-default L1) — `benchmarks/results/bench_e2e_n16_20260918-032258.json` |
 | p150a (served) | this image, `POST /predict` on the demo observation, seed path, 50 requests after 5 warm-ups (`code/scripts/bench_http.py`), server-side `timing_ms` |
 
 ## Correctness reference (what "same output" means here)
@@ -24,16 +26,18 @@ The port is gated against the **fp32 CPU run** of NVIDIA's code (the golden). NV
 sits at a measurable distance from that golden, which bounds what any bf16 implementation can be asked to reproduce
 (`code/models/experimental/gr00t/common/golden_tap_map.json`, `thresholds.n16`):
 
-| tensor | official bf16 GPU vs fp32 golden (PCC) | p150a Stage-1 vs fp32 golden (PCC) | gate |
-|---|---:|---:|---:|
-| `backbone_features` (LLM output, 116 tokens × 2048) | 0.99679 | 0.99878 | 0.9917 |
-| `action_pred_valid` (16 steps × 29 action dims, normalised) | 0.99956 | 0.99921 | 0.998 |
-| `unnormalized_actions` — `left_arm` / `right_arm` / `right_hand` | — | 0.99995 / 0.99991 / 0.99312 | 0.9999 / 0.9998 / 0.99 |
+| tensor | official bf16 GPU vs fp32 golden (PCC) | p150a Stage-2 (megakernel) vs fp32 golden (PCC) | p150a Stage-1 image (2026-09-15) | gate |
+|---|---:|---:|---:|---:|
+| `backbone_features` (LLM output, 116 tokens × 2048) | 0.99679 | 0.99878 | 0.99878 | 0.9917 |
+| `action_pred_valid` (16 steps × 29 action dims, normalised) | 0.99956 | 0.99922 | 0.99921 | 0.998 |
+| `unnormalized_actions` — `left_arm` / `right_arm` / `right_hand` | — | 0.99995 / 0.99990 / 0.99308 | 0.99995 / 0.99991 / 0.99312 | 0.9999 / 0.9998 / 0.99 |
 
-p150a values: `tests/tt/results/test_e2e_{untraced_taps_vs_golden,predict_actions_vs_golden}_n16_20260914-2324*.json`. Four
-intermediate DiT taps of the p150a run miss their per-tap gates on the unused state-token row (`dit_out[k=1]` 0.999556 vs 0.99961,
-`dit_out[k=2]` 0.999496 vs 0.99964, `action_decoder_out[k=1]` 0.999579 vs 0.9997, `action_decoder_out[k=2]` 0.999556 vs 0.99978);
-every action tap passes — see the card's Caveats.
+p150a Stage-2 values: `tests/tt/results/test_e2e_{untraced_taps_vs_golden,predict_actions_vs_golden}_n16_20260918-0450*.json`
+(the snapshot validation). Two intermediate taps of the megakernel path miss their per-tap gates on the unused state-token row
+(`action_decoder_out[k=1]` 0.999594 vs 0.9997, `action_decoder_out[k=2]` 0.999598 vs 0.99978; 26/28 gated rows pass); the DiT block
+intermediates (`dit_out[k]`, two of which missed on the Stage-1 path) are not observable inside the megakernel; every action tap
+passes — see the card's Caveats. Megakernel vs TTNN denoise on device (same tree, `mk_default_summary.md` §3.1): `action_pred_normalized`
+PCC ≥ 0.999994 on every N1.6 reference sample.
 
 ## GPU latency (RTX 5090, batch 1, medians of 20 iterations, ms)
 
@@ -48,43 +52,49 @@ every action tap passes — see the card's Caveats.
 The official rows reproduce NVIDIA's README table for the RTX 5090 (eager 58 ms, compiled 37 ms e2e; backbone 18 ms; action head
 38 → 16 ms). The "golden" rows are the settings the fp32 goldens were dumped with; the deterministic mode costs ~35 ms here;
 on N1.7 the same `CUBLAS_WORKSPACE_CONFIG=:4096:8` env var alone was shown to double the small-M DiT GEMMs
-(`docs/research/golden-multisample.md` §3) — not separately attributed for N1.6 — and it is not a serving configuration. The eager action head is dispatch-bound (128 DiT-block evaluations in 36 ms), which is why `torch.compile` gains 2.7×
-on it — and the regime the port's Stage-2 persistent megakernel targets.
+(`docs/research/golden-multisample.md` §3) — not separately attributed for N1.6 — and it is not a serving configuration. The eager action
+head is dispatch-bound (128 DiT-block evaluations in 36 ms), which is why `torch.compile` gains 2.7× on it — the regime the p150a's
+persistent megakernel addresses with one launch per denoise.
 
 ## p150a latency (batch 1, ms)
 
-| row | median | p90 | source |
-|---|---:|---:|---|
-| `bench_e2e` end-to-end (host encode + upload + 4 traces + D2H + host decode) | **59.42** | 60.10 | `e2e_stage1_n16.json` `stages.e2e` |
-| device = upload + traces + D2H | 57.20 | 57.67 | `stages.device` |
-| 4 traces (one sync) | 53.27 | 53.29 | `stages.traces` |
-| trace `vision` (SigLIP2 tower + unshuffle projector) | 11.49 | 11.50 | `stages.trace/vision` (split pass) |
-| trace `llm` (Qwen3-1.7B[:16], S 116 → 128) | 9.50 | 9.52 | `stages.trace/llm` |
-| trace `adapter` (VL-LN, K/V hoist, state encoder) | 1.98 | 1.99 | `stages.trace/adapter` |
-| trace `denoise` (4 Euler steps × 32 DiT blocks + decoder) | 30.40 | 30.41 | `stages.trace/denoise` |
-| upload / D2H / host encode / host decode | 3.79 / 0.09 / 1.99 / 0.17 | 4.31 / 0.11 / 2.28 / 0.21 | `stages.*` |
-| served `timing_ms.device` (this image, 50 requests after 5 warm-ups, quiet host) | 56.8 | 57.3 | `bench_http.py` run of the container validation (round 1, `bench-n16-c1-20260915-014844.json`) |
-| served `timing_ms.total` (handler wall incl. base64 + PNG decode) | 59.4 | 59.8 | same |
-| served client wall (localhost, incl. HTTP + JSON) | 61.0 | 61.6 | same |
+| row | Stage 2 (this image): median | p90 | TTNN denoise, same tree: median | p90 | source |
+|---|---:|---:|---:|---:|---|
+| `bench_e2e` end-to-end (host encode + upload + 4 traces + D2H + host decode) | **40.67** | 40.77 | 56.20 | 56.60 | `stages.e2e` of `bench_e2e_n16_20260918-050632.json` / `…-032258.json` |
+| device = upload + traces + D2H | 39.17 | 39.22 | 54.39 | 54.77 | `stages.device` |
+| 4 traces (one sync) | 37.76 | 37.78 | 52.65 | 52.70 | `stages.traces` |
+| trace `vision` (SigLIP2 tower + unshuffle projector) | 10.80 | 11.00 | 10.82 | 10.87 | `stages.trace/vision` (split pass) |
+| trace `llm` (Qwen3-1.7B[:16], S 116 → 128) | 9.55 | 9.75 | 9.55 | 9.61 | `stages.trace/llm` |
+| trace `adapter` (VL-LN, K/V hoist, state encoder; intermediates in DRAM under the megakernel) | 2.24 | 2.44 | 2.01 | 2.07 | `stages.trace/adapter` |
+| trace `denoise` (4 Euler steps × 32 DiT blocks + decoder) | **15.35** (3.84 per step) | 15.52 | 30.43 | 30.53 | `stages.trace/denoise` |
+| upload / D2H / host encode / host decode | 1.38 / 0.04 / 1.41 / 0.10 | 1.41 / 0.05 / 1.48 / 0.10 | 1.66 / 0.07 / 1.61 / 0.17 | 1.94 / 0.09 / 1.79 / 0.23 | `stages.*` |
+| served `timing_ms.device` (this image, 50 requests after 5 warm-ups) | 39.6 | 39.9 | 56.8 (Stage-1 image `feb77048a2ed`) | 57.3 | `bench_http.py` run of the container validation (`bench-mk-n16-c1-20260918-054614.json`; Stage-1: `bench-n16-c1-20260915-014844.json`) |
+| served `timing_ms.total` (handler wall incl. base64 + PNG decode) | 42.1 | 42.5 | 59.4 | 59.8 | same |
+| served client wall (localhost, incl. HTTP + JSON) | 44.3 | 44.8 | 61.0 | 61.6 | same |
+
+The Stage-1 **release** of 2026-09-15 measured 59.42 ms e2e (`benchmarks/results/e2e_stage1_n16.json`: 1 command queue, host embedding
+gather); the same-tree TTNN column above already carries the 2-CQ uploads and the device embedding gather that the tree gained since, so
+the megakernel's own effect is the difference between the two columns: −15.5 ms e2e (−27.6 %), all of it in the `denoise` trace
+(−15.1 ms, −49.6 %) minus the adapter's DRAM placement (+0.2 ms).
 
 ## Comparison with matching definitions
 
 Ratio = GPU ms / p150a ms (> 1 means the p150a is faster).
 
-| row | p150a | GPU | ratio |
+| row | p150a (Stage 2) | GPU | ratio |
 |---|---:|---:|---:|
-| end-to-end policy call: `bench_e2e` e2e 59.42 (encode + device + decode) vs `get_action` eager bf16 e2e 63.1 (data proc + backbone + head) | 59.42 | 63.1 | **1.06** (p150a faster) |
-| same vs `get_action` with `torch.compile(DiT)` | 59.42 | 33.1 | **0.56** (GPU 1.80× faster) |
-| model only: p150a device 57.20 (upload + traces + D2H) vs GPU backbone + head 53.0 (eager) | 57.20 | 53.0 | 0.93 (GPU 1.08× faster) |
-| model only vs GPU compiled backbone + head 30.6 | 57.20 | 30.6 | 0.53 (GPU 1.87× faster) |
-| denoise: p150a `denoise` trace 30.40 vs GPU action head 36.2 (eager) / 13.5 (compiled) | 30.40 | 36.2 / 13.5 | 1.19 / 0.44 |
-| vision + LLM: p150a `vision` + `llm` traces 20.99 vs GPU backbone 16.8 | 20.99 | 16.8 | 0.80 (GPU 1.25× faster) |
-
-Reading: against NVIDIA's eager bf16 path the Stage-1 p150a port is at parity end-to-end (1.06× faster); against the compiled
-DiT the GPU is 1.8× faster, entirely in the denoise stage (30.4 vs 13.5 ms) — the p150a's four-step DiT is the part the
-persistent megakernel (Stage 2, not in this image; weight streaming measured at 464 GB/s bf16 / 414 GB/s bfp8 in
-`tests/tt/results/mk_k1_summary.md`) is meant to replace. The GPU's `data proc` (2.5 ms) and the p150a's `host encode` (2.0 ms) are the
-same kind of host work; HTTP/JSON framing is excluded on both sides except in the "served client wall" row.
+| end-to-end policy call: `bench_e2e` e2e 40.67 (encode + device + decode) vs `get_action` eager bf16 e2e 63.1 (data proc + backbone + head) | 40.67 | 63.1 | **1.55** (p150a faster) |
+| same vs `get_action` with `torch.compile(DiT)` | 40.67 | 33.1 | **0.81** (GPU 1.23× faster) |
+| model only: p150a device 39.17 (upload + traces + D2H) vs GPU backbone + head 53.0 (eager) | 39.17 | 53.0 | 1.35 (p150a faster) |
+| model only vs GPU compiled backbone + head 30.6 | 39.17 | 30.6 | 0.78 (GPU 1.28× faster) |
+| denoise: p150a `denoise` trace 15.35 vs GPU action head 36.2 (eager) / 13.5 (compiled) | 15.35 | 36.2 / 13.5 | 2.36 / 0.88 (GPU 1.14× faster) |
+| vision + LLM: p150a `vision` + `llm` traces 20.35 vs GPU backbone 16.8 | 20.35 | 16.8 | 0.83 (GPU 1.21× faster) |
+
+Reading: against NVIDIA's eager bf16 path the Stage-2 p150a port is 1.55× faster end-to-end (the Stage-1 release was 1.06×); against
+the compiled DiT the GPU is still 1.23× faster (1.80× against Stage 1). The denoise gap closed from 30.4 vs 13.5 ms to 15.35 vs 13.5 ms;
+the remaining e2e gap is now the backbone (`vision` + `llm` 20.35 ms vs 16.8 ms on the GPU) plus the p150a's upload / readback, not the
+DiT. The GPU's `data proc` (2.5 ms) and the p150a's `host encode` (1.4 ms) are the same kind of host work; HTTP/JSON framing is excluded on
+both sides except in the "served client wall" row.
 
 Not measured / not claimed: p150a power (no power or efficiency comparison); TensorRT or FP8 on the GPU; the compiled GPU path with
 the p150a's exact host chain; N1.6's `gr1_long_text` layout. GPU numbers were not re-run for this card; they are copied with their

LICENSE-NOTICE.md

@@ -9,7 +9,7 @@ and a card. It contains **no model weights**. Four licences apply to what is her
 | Vendored tokenizer files `code/gr00t_p150/assets/tokenizer/n16/{vocab.json, merges.txt, tokenizer_config.json, special_tokens_map.json, added_tokens.json}` | in this repo and in the image (`/opt/tt-metal/gr00t_p150/assets/tokenizer/n16`) | copied from [NVIDIA/Isaac-GR00T](https://github.com/NVIDIA/Isaac-GR00T) tag `n1.6.1-release`, path `gr00t/model/modules/nvidia/Eagle-Block2A-2B-v2/`; that repository's `LICENSE` is the same **NVIDIA License** as the weights. The files are the Qwen2 BPE vocabulary + merges plus the Eagle special tokens (`<IMG_CONTEXT>` = 151669, `<img>` = 151670, `</img>` = 151671); they are not part of the weights repo, which is why they ship here. |
 | Demo observation `media/demo_ego_view.png`, `media/demo_observation.json`, `code/gr00t_p150/demo/n16/` (frame, raw state, instruction, the reference initial noise and the fp32 / bf16 reference actions) | in this repo and in the image | one GR1 PickNPlace step from the Isaac-GR00T demo data (`demo_data`, distributed with the Isaac-GR00T repository); the reference actions were computed with NVIDIA's unmodified code on the pinned weights and are therefore derivative of the weights (NVIDIA License §3.2 applies to their use). |
 | Port and serving code: `code/models/experimental/gr00t/` (the tt-nn port of GR00T N1.5 / N1.6 / N1.7: `common/`, `reference/`, `tt/` incl. `tt/megakernel/`, `tests/`, `benchmarks/`), `code/gr00t_p150/` (FastAPI server, schemas, smoke test, demo builder), `code/scripts/` | in this repo and in the image (`/opt/tt-metal/...`) | **Apache-2.0** (`SPDX-License-Identifier: Apache-2.0`, © 2026 Tenstorrent USA, Inc. in every source file). Because the port only runs the NVIDIA-licensed weights and is distributed together with the NVIDIA-licensed tokenizer files and reference actions, the package as a whole is offered under the NVIDIA License's non-commercial use limitation; the Apache-2.0 grant on the code itself is unaffected. |
-| tt-metal / tt-nn: `code/models/common/utility_functions.py`, `code/models/tt_dit/**`, `code/models/demos/deepseek_v3_b1/unified_kernel_descriptor.py` and the whole tt-metal build inside the image (`/opt/tt-metal`, commit `668c290757550588d0ce46b180c344a462a2aaf5`) | in this repo (the three files) and in the image | **Apache-2.0** ([tenstorrent/tt-metal](https://github.com/tenstorrent/tt-metal/blob/main/LICENSE)). |
+| tt-metal / tt-nn: `code/models/common/utility_functions.py`, `code/models/tt_dit/**`, `code/models/demos/deepseek_v3_b1/unified_kernel_descriptor.py`, `code/models/demos/deepseek_v3_b1/unified_kernels/{kernel_op_api,kernel_utils,dataflow_utils,mcast}.hpp` (device-code headers the megakernel includes) and the whole tt-metal build inside the image (`/opt/tt-metal`, commit `668c290757550588d0ce46b180c344a462a2aaf5`) | in this repo (the tree-staged files under `code/models/{common,tt_dit,demos}`) and in the image | **Apache-2.0** ([tenstorrent/tt-metal](https://github.com/tenstorrent/tt-metal/blob/main/LICENSE)). |
 | Third-party Python packages inside the image (`requirements.lock`: torch 2.11.0+cpu, transformers 5.12.1, opencv-python-headless 4.8.1.78, numpy 1.26.4, fastapi, uvicorn, pydantic, pillow, safetensors, huggingface_hub, loguru, ...) and the Ubuntu 22.04 base | in the image | their respective licences (BSD-3 / Apache-2.0 / MIT / HPND; Ubuntu package licences). |
 
 ## Practical reading

README.md

@@ -4,11 +4,24 @@ tags:
 - p150
 - tt-dit-server
 - tt-model-cache
-- tt-model-catalog
 - tt-model-container
+- tenstorrent
+- ttnn
+- tt-metal
+- tt-nn
+- robotics
+- vla
+- gr00t
+- tt-model-catalog
+pipeline_tag: robotics
+license: other
+license_name: nvidia-license
+license_link: https://huggingface.co/nvidia/GR00T-N1.6-3B/blob/main/LICENSE
+base_model:
+- nvidia/GR00T-N1.6-3B
 ---
 
-# gr00t-n1.6-3b-p150
+# GR00T-N1.6-3B-p150
 
 NVIDIA Isaac GR00T N1.6 (3B vision-language-action policy) running on one Tenstorrent Blackhole p150a via tt-nn: one camera frame, proprioceptive state and a task instruction in, a 16-step chunk of GR1 arm, hand and waist actions out. Stage-2 release: the whole DiT denoise (32 blocks × 4 flow-matching steps) runs as one persistent megakernel streaming bfp8 weights from a DRAM arena; vision, LLM and adapter are TTNN ops (bf16) replayed from Metal traces; `GR00T_DIT_BACKEND=ttnn` selects the Stage-1 denoise.
 Weights: [nvidia/GR00T-N1.6-3B](https://huggingface.co/nvidia/GR00T-N1.6-3B) · Paper: [arXiv:2503.14734](https://arxiv.org/abs/2503.14734) · Project page: [gr00t-n1_6](https://research.nvidia.com/labs/gear/gr00t-n1_6/) · Upstream code: [NVIDIA/Isaac-GR00T](https://github.com/NVIDIA/Isaac-GR00T) (`n1.6.1-release`) · Port: `code/models/experimental/gr00t` and the server `code/gr00t_p150` in this repo
@@ -24,7 +37,8 @@ tt-model pull  changh95/GR00T-N1.6-3B-p150 --with-weights
 tt-model serve changh95/GR00T-N1.6-3B-p150
 ```
 
-`pull --with-weights` downloads the Docker image and the [`nvidia/GR00T-N1.6-3B`](https://huggingface.co/nvidia/GR00T-N1.6-3B) weights at `d0814e7ecb19202e7c8468b46098b0b7ef3a6d61` (into your HF cache; they are not in the image). `serve` starts the model's own HTTP server on port 20000 (or the next free port, if that one is busy); the first start compiles kernels for your device, which takes several minutes, and the server is ready when it logs `Application startup complete`.
+- Weights [`nvidia/GR00T-N1.6-3B`](https://huggingface.co/nvidia/GR00T-N1.6-3B) at `d0814e7ecb19` (6.57 GB) go to your HF cache; the image does not contain them.
+- Serves on port 20000 (or the next free port); ready when the log says `Application startup complete`.
 
 ### Run with tt-cli
 

SERVING.md

@@ -3,17 +3,18 @@
 This repo is a **tt-model container package source**: `tt-model.yaml` + the code under `code/`. `tt-model package
 --container` builds the OCI image (tt-metal from source + the port + the HTTP stack), `tt-model serve` runs it against
 the card, `tt-model push` publishes it here. The server is `code/gr00t_p150/server/app.py` (FastAPI, `kind:
-tt-dit-server`); the device model is the tt-nn port `code/models/experimental/gr00t` (Stage-1 path: TTNN ops captured
-in four Metal traces, DiT matmul weights bfp8_b, everything else bf16). Weights are a pinned pointer, never in the image.
+tt-dit-server`); the device model is the tt-nn port `code/models/experimental/gr00t` (Stage-2 path: vision / LLM / adapter
+as TTNN ops captured in three Metal traces, the whole DiT denoise as **one persistent megakernel `generic_op`** in the
+fourth trace, streaming its weights from a bfp8_b DRAM arena; everything else bf16). Weights are a pinned pointer, never in the image.
 
 | | |
 |---|---|
 | tt-metal | main `668c290757550588d0ce46b180c344a462a2aaf5` (`v0.79.0-dev20260914-1-g668c290757`), built from source inside the image; torch pin 2.11.0 (+cpu in the image) |
 | weights | `nvidia/GR00T-N1.6-3B` @ `d0814e7ecb19202e7c8468b46098b0b7ef3a6d61` — `model-0000{1,2}-of-00002.safetensors` (6,573,217,664 B of bf16 tensors; 6,573,374,552 B on disk in 2 files), `model.safetensors.index.json`, `config.json`, `processor_config.json`, `statistics.json`, `embodiment_id.json`, `LICENSE` (NVIDIA License, non-commercial) |
-| app | `gr00t_p150.server.app:app` (uvicorn `--lifespan on`; the lifespan resolves the snapshot, opens the device, builds `Gr00tTT`, captures the traces, runs `GR00T_WARMUP_RUNS` traced predictions and checks them against the shipped fp32 golden before READY) |
-| device recipe | `models.experimental.gr00t.tt.device.open_gr00t_device(device_id, trace_region_size=64 MiB, l1_small_size=32768, num_command_queues=1)` — the port's validated parameters; asserts the 11×10 compute / 8×1 DRAM grid of a p150 |
-| port source | `code/models/experimental/gr00t` = the frozen snapshot commit `5dc0517ee3bec090b7ead025001a343c2f63aa7b` (ref `gr00t-p150-snapshot`, parent = `668c2907575`) of the tt-metal working tree `models/experimental/gr00t`; `code/gr00t_p150` (server) and `code/scripts` were written for this release |
-| only served configuration | version `n16`, embodiment `gr1` (slot 20), static layout `gr1` (1 camera × 81 image tokens, `L_max` 24 text tokens, LLM sequence padded to 128), `mixed_dit` dtype policy, `per_stage` trace layout, batch 1 |
+| app | `gr00t_p150.server.app:app` (uvicorn `--lifespan on`; the lifespan resolves the snapshot, resolves the `TTPolicy`, opens the device for it, builds `Gr00tTT`, captures the traces, runs `GR00T_WARMUP_RUNS` traced predictions and checks them against the shipped fp32 golden before READY) |
+| device recipe | `models.experimental.gr00t.tt.model.open_model_device(policy, device_id, trace_region_size=64 MiB, l1_small_size=32768, num_command_queues=2)`: for `dit_backend=megakernel` the device is opened with `worker_l1_size` 1,461,248 − 65,536 = **1,395,712 B** (a 136,192 B kernel-config ring for the megakernel binaries), program cache on; asserts the 11×10 compute / 8×1 DRAM grid of a p150. `GR00T_DIT_BACKEND=ttnn` opens the firmware-default L1 |
+| port source | `code/models/experimental/gr00t` = the frozen snapshot commit `009de892c895917ad6abe9a6c36518f00316ed4b` (ref `gr00t-p150-snapshot-mk`, parent = `668c2907575`) of the tt-metal working tree `models/experimental/gr00t`; megakernel tree `tt/megakernel/**` = the K4b-final checkpoint (`tests/tt/results/mk_tree_20260918-041439.sha256`, digest `7df61c35e9dc7857`); `code/gr00t_p150` (server) and `code/scripts` were written for this release |
+| only served configuration | version `n16`, embodiment `gr1` (slot 20), static layout `gr1` (1 camera × 81 image tokens, `L_max` 24 text tokens, LLM sequence padded to 128), `mixed_dit` dtype policy, `per_stage` trace layout, `dit_backend` **`megakernel`** with the `bfp8_b` arena, 2 command queues, batch 1 |
 
 Below, `$ROOT` = the tt-models tooling checkout (`/home/deepgadget/experiments/tt-models` on the build host: `tt-model` CLI in
 `$ROOT/.venv/bin`, rootless Docker env in `$ROOT/bin/docker-env.sh`), `$TREE` = the tt-metal tree with the built `python_env`
@@ -31,23 +32,27 @@ SERVING.md, GPU_COMPARISON.md, LICENSE, LICENSE-NOTICE.md     authored
 media/demo_ego_view.png       the GR1 PickNPlace demo frame (256x256), media/demo_observation.json the matching raw state + instruction
 media/demo_actions_vs_golden.png   served action chunk vs the fp32 reference, one panel per action dimension
 code/models/experimental/gr00t/    the port: common/ (host preprocessing, prompts, normalisation, checkpoint, weight plan),
-                                   reference/ (fp32 torch reference), tt/ (device model, layers, traces, tt/megakernel/ Stage-2 work),
-                                   tests/ (CPU + device suites and their results JSONs), benchmarks/ (bench_e2e etc. + results)
-code/models/common/utility_functions.py, code/models/tt_dit/{utils,layers,parallel,encoders/qwen3vl}/..., code/models/demos/deepseek_v3_b1/unified_kernel_descriptor.py
-                                   the tt-metal files the port imports from the tree other than itself (staged by tt-model from `source.code`)
+                                   reference/ (fp32 torch reference), tt/ (device model, layers, traces), tt/megakernel/ (the DiT
+                                   megakernel: arena, core map, descriptors, dit_program.py, kernels/*.cpp|hpp), tests/ (CPU + device
+                                   suites; tests/tt/results/ = the summaries + the snapshot-validation JSONs), benchmarks/ (bench_e2e etc. + results)
+code/models/common/utility_functions.py, code/models/tt_dit/{utils,layers,parallel,encoders/qwen3vl}/..., code/models/demos/deepseek_v3_b1/{unified_kernel_descriptor.py, unified_kernels/{kernel_op_api,kernel_utils,dataflow_utils,mcast}.hpp}
+                                   the tt-metal files the port imports from the tree other than itself (staged by tt-model from `source.code`; the four
+                                   .hpp are device-code headers dit_kernel.cpp includes, JIT-compiled at boot)
 code/gr00t_p150/              server/{app.py, schemas.py, smoke_test.py}, demo/n16/ (canonical demo observation + golden), assets/tokenizer/n16/ (vendored Qwen2 tokenizer)
 code/scripts/                 download_weights.sh (pinned snapshot -> HF cache), bench_http.py (HTTP latency, 50 requests)
 ```
 
-In the image the code lands at `/opt/tt-metal/<path>` with `PYTHONPATH=/opt/tt-metal`, which is exactly the tree layout the port was
-written for (`models.experimental.gr00t.*`, `models.tt_dit.*` absolute imports; `models` is a namespace package).
+In the image the code lands at `/opt/tt-metal/<path>` with `PYTHONPATH=/opt/tt-metal` and `TT_METAL_HOME=/opt/tt-metal`, which is
+exactly the tree layout the port was written for (`models.experimental.gr00t.*`, `models.tt_dit.*` absolute imports; `models` is a
+namespace package; the megakernel source `models/experimental/gr00t/tt/megakernel/kernels/dit_kernel.cpp` is JIT-compiled by tt-metal
+relative to `TT_METAL_HOME`).
 
 ## 1. Run on the HOST (hardware validation, no Docker)
 
 The tree venv `$TREE/python_env` (Python 3.10, torch 2.11.0+cpu, ttnn editable, transformers 5.12.1, opencv 4.8.1) has everything
 except `fastapi` / `uvicorn`. Do not install into the tree venv; put the HTTP stack in a side directory and prepend it to `PYTHONPATH`.
 `$REPO/code` must be FIRST on `PYTHONPATH` so `models.experimental.gr00t` resolves to the shipped snapshot (the tree's `models/` is still
-needed for `models.tt_dit.*` on the host; in the image the staged `code/` is the only `models` tree).
+needed for `models.tt_dit.*` / `models.demos.*` on the host; in the image the staged `code/` is the only `models` tree).
 
 ```bash
 export TREE=/home/deepgadget/experiments/gr00t/tt-metal
@@ -58,20 +63,22 @@ uv pip install --python $TREE/python_env/bin/python --target $HTTP fastapi uvico
 cd $REPO
 export PYTHONPATH=$PWD/code:$TREE:$TREE/ttnn:$TREE/tools:$HTTP
 export TT_METAL_HOME=$TREE ARCH_NAME=blackhole
-export TT_METAL_CACHE=$HOME/.cache/tt-metal-cache-gr00t                  # JIT kernels (7 GB warm on the build host)
+export TT_METAL_CACHE=$HOME/.cache/tt-metal-cache-gr00t                  # JIT kernels (incl. the megakernel binaries)
 export HF_MODEL=nvidia/GR00T-N1.6-3B
 export TT_WEIGHTS_REVISION=d0814e7ecb19202e7c8468b46098b0b7ef3a6d61      # scripts/download_weights.sh puts it in ~/.cache/huggingface
 export TT_MESH_SHAPE=1x1 TT_DEVICE_ID=0 TT_METAL_VISIBLE_DEVICES=0
 export GR00T_VERSION=n16 GR00T_EMBODIMENT=gr1 GR00T_LAYOUT=gr1 GR00T_POLICY=mixed_dit GR00T_TRACE_LAYOUT=per_stage
+export GR00T_DIT_BACKEND=megakernel GR00T_MK_ARENA_DTYPE=auto GR00T_NUM_CQS=2   # the image's serve.env; unset = the same port defaults
 export GR00T_TOKENIZER_DIR=$PWD/code/gr00t_p150/assets/tokenizer/n16
-export GR00T_TT_CACHE=$HOME/.cache/gr00t-tt                              # host .pt plan tier + .tensorbin device tier (~8 GB for n16)
+export GR00T_TT_CACHE=$HOME/.cache/gr00t-tt                              # host .pt plan tier + .tensorbin device tier (~2.6 GB under the megakernel for n16; 7.9 GB on the Stage-1 path)
 export GR00T_WARMUP_RUNS=3
 
 # import check, no device (what the image's verify.sh does):
 $TREE/python_env/bin/python -c "import gr00t_p150.server.app as a; assert a.app; print('ok')"
 
-# serve (opens the device, loads the weights, captures the four traces, 3 warm-up predictions checked against the golden, then READY).
-# The one Blackhole card of the build host is shared with other agents: run server + client + shutdown inside ONE lock hold.
+# serve (opens the device for the policy, loads the weights, builds the megakernel arena, captures the four traces,
+# 3 warm-up predictions checked against the golden, then READY). The one Blackhole card of the build host is shared
+# with other agents: run server + client + shutdown inside ONE lock hold.
 DEVICE_LOCK_TIMEOUT=14400 /home/deepgadget/experiments/gr00t/bin/with-device.sh bash -c '
   $TREE/python_env/bin/python -m uvicorn --host 127.0.0.1 --port 20016 --lifespan on gr00t_p150.server.app:app & UV=$!
   python3 code/gr00t_p150/server/smoke_test.py --url http://127.0.0.1:20016 --wait 1800 --out /tmp/gr00t-n16-smoke.json; RC=$?
@@ -80,22 +87,35 @@ DEVICE_LOCK_TIMEOUT=14400 /home/deepgadget/experiments/gr00t/bin/with-device.sh
 ```
 
 Boot log landmarks (they drive `tt-model serve`'s boot checklist): `Loading weights: nvidia/GR00T-N1.6-3B @ d0814e7e…` →
-`Tokenizer files: …` → `Opening device 0 (trace_region_size=67108864, l1_small_size=32768)` → `Loading pipeline: Gr00tTT.from_pretrained(...)`
-→ `Model built in N s (1037 tensors, 3775 MB on device, cache path warm|cold)` → `Warming up: capture per_stage traces on the n16 demo
-observation, then 3 traced predict(s)` → `Warmup k/3: … ms` → `Warm-up fidelity vs golden fp32: actions PCC 0.9992…` → `Warmup complete (…)`
-→ uvicorn `Application startup complete`. Startup failures raise and uvicorn exits non-zero (no CPU fallback); a warm-up PCC below
-`GR00T_WARMUP_PCC_MIN` (0.999) fails the boot. Stop with SIGTERM / Ctrl-C: the lifespan releases the traces and device tensors and closes
-the device (`Releasing traces and device tensors`, `Closing device`).
-
-Expected smoke line on the demo observation (host, warm caches, `publish/logs/smoke_n16_snapshot_20260914-235724.log` of the build host):
+`Tokenizer files: …` → `Opening device 0 (trace_region_size=67108864, l1_small_size=32768, num_command_queues=2, dit_backend=megakernel,
+mk_arena_dtype=bfp8_b)` → `Loading pipeline: Gr00tTT.from_pretrained(...)` → `Model built in N s (weights …, 503 tensors, 2744 MB on device,
+cache path warm|cold; dit_backend megakernel, arena bfp8_b, adapter_mem DRAM, …; ttnn DiT weights skipped 534 tensors / 1032 MB)` →
+`Warming up: capture per_stage traces on the n16 demo observation, then 3 traced predict(s)` → `Warmup k/3: … ms` →
+`Warm-up fidelity vs golden fp32: actions PCC 0.99923…` → `Warmup complete (…)` → uvicorn `Application startup complete`. Startup
+failures raise and uvicorn exits non-zero (no CPU fallback); a warm-up PCC below `GR00T_WARMUP_PCC_MIN` (0.999) fails the boot. Stop with
+SIGTERM / Ctrl-C: the lifespan releases the traces and device tensors and closes the device (`Releasing traces and device tensors`,
+`Closing device`).
+
+Expected smoke line on the demo observation (host, live tree at the snapshot state, **cold** `TT_METAL_CACHE`, warm weight tiers;
+`publish/logs/smoke_n16_20260918-043318.log` of the build host, `tests/tt/results/mk_default_summary.md` §7):
 
 ```
-PASS GR00T-N1.6-3B-p150 n16: PCC(actions)=0.999209 max|d|=0.0971 PCC(action_pred_valid)=0.999213 repeat_maxdiff=0.0e+00 seed_path_ok device_ms=57.14/57.03 total_ms=64.11/62.68 wall_ms=70 per_group[left_arm=0.99995/0.0176 right_arm=0.99991/0.0423 left_hand=0.80114/0.0589 right_hand=0.99312/0.0971 waist=0.77216/0.0170]
+PASS GR00T-N1.6-3B-p150 n16: PCC(actions)=0.999232 max|d|=0.0925 PCC(action_pred_valid)=0.999218 repeat_maxdiff=0.0e+00 seed_path_ok device_ms=40.21/39.37 total_ms=47.94/44.63 wall_ms=55 per_group[left_arm=0.99995/0.0176 right_arm=0.99990/0.0423 left_hand=0.79648/0.0589 right_hand=0.99308/0.0925 waist=0.78038/0.0134]
 ```
 
-Host boot with warm `GR00T_TT_CACHE` + `TT_METAL_CACHE`: model built 0.6–8.5 s (page cache), capture 1.7–2.0 s, 5 warm-ups ~60 ms each,
-READY after 5–13 s. Offline overrides: `GR00T_WEIGHTS_DIR=<dir with the seven snapshot files>` (shimmed into a private hub cache so the
-port's `configs.snapshot_dir("n16")` resolves to it); `GR00T_TOKENIZER_DIR=<dir>`. Every variable is read in the lifespan, never at import.
+That host boot (every kernel JIT-compiled, incl. the megakernel): model built 4.3 s, `warm_and_capture` 33.1 s, 5 warm-ups 41.4 ms, READY
+after 41.2 s (`/info -> warmup_latency_ms`). Offline overrides: `GR00T_WEIGHTS_DIR=<dir with the seven snapshot files>` (shimmed into a
+private hub cache so the port's `configs.snapshot_dir("n16")` resolves to it); `GR00T_TOKENIZER_DIR=<dir>`. Every variable is read in
+the lifespan, never at import.
+
+Environment knobs of the Stage-2 server (all read in the lifespan; the image's `serve.env` sets the first three explicitly):
+
+| variable | image value | meaning |
+|---|---|---|
+| `GR00T_DIT_BACKEND` | `megakernel` | `TTPolicy.dit_backend`: `megakernel` (the DiT denoise as one persistent `generic_op`, Stage 2, the port's default) or `ttnn` (the Stage-1 TTNN op sequence, for A/B on the same image). Reported in `/info -> stage.served` (`stage2-megakernel` / `stage1-ttnn-traces`) |
+| `GR00T_MK_ARENA_DTYPE` | `auto` (-> `bfp8_b`) | `TTPolicy.mk_arena_dtype` of the megakernel's DRAM weight arena: `auto` / `bf16` / `bfp8_b`; the resolved value is `/info -> stage.mk_arena_dtype` |
+| `GR00T_NUM_CQS` | `2` | command queues the device is opened with: `2` = per-call input writes on CQ 1 overlapping the traces (bit-equal to `1` on every probe, `tests/tt/results/mk_default_summary.md` §4) |
+| `GR00T_WARMUP_PCC_MIN` | `0.999` | boot self-check on the warm-up chunk vs the shipped fp32 golden; the served N1.6 chunk scores 0.99923 |
 
 ## 2. Package, serve, push (Docker)
 
@@ -109,7 +129,7 @@ cd $REPO
 # offline validation (must print VALID)
 $ROOT/.venv/bin/python -c "from tt_kernel.container_manifest import load_container_manifest as L; m = L('tt-model.yaml', check_sources=True); print('VALID', m.name, m.weights_ref)"
 
-$ROOT/.venv/bin/tt-model package --container tt-model.yaml --out /home/deepgadget/experiments/gr00t/publish/build   # ~2 h cold (tt-metal C++ build), runs verify.sh
+$ROOT/.venv/bin/tt-model package --container tt-model.yaml --out /home/deepgadget/experiments/gr00t/publish/build   # minutes with the ccache mount (~2 h cold), runs verify.sh
 # serve -> smoke -> bench -> stop inside ONE device-lock hold (the container keeps the card after `serve` returns)
 DEVICE_LOCK_TIMEOUT=14400 /home/deepgadget/experiments/gr00t/bin/with-device.sh bash -c '
   source $ROOT/bin/docker-env.sh
@@ -125,37 +145,35 @@ $ROOT/.venv/bin/tt-model push /home/deepgadget/experiments/gr00t/publish/build/g
 -v ~/.cache/tt-model/gr00t-n1.6-3b-p150/{cache,weights,tensors}:/{cache,weight-cache,tensor-cache} -p 20000:20000` with exactly
 `HF_MODEL=nvidia/GR00T-N1.6-3B`, `MESH_DEVICE=P150`, `TT_MESH_SHAPE=1x1`, `HF_HOME=/hf`, `TT_METAL_CACHE=/cache`,
 `TT_DIT_CACHE_DIR=/weight-cache`, `TT_CACHE_PATH=/tensor-cache` and the `serve.env` block of `tt-model.yaml`
-(`TT_WEIGHTS_REVISION`, `GR00T_VERSION=n16`, `GR00T_TT_CACHE=/weight-cache/gr00t-tt`, `GR00T_TOKENIZER_DIR=/opt/tt-metal/gr00t_p150/assets/tokenizer/n16`, …),
-and waits ≤ 1800 s for `Application startup complete`. Serves on 20000 or the next free port (printed). tt-cli users after the push:
+(`TT_WEIGHTS_REVISION`, `GR00T_VERSION=n16`, `GR00T_DIT_BACKEND=megakernel`, `GR00T_MK_ARENA_DTYPE=auto`, `GR00T_NUM_CQS=2`,
+`GR00T_TT_CACHE=/weight-cache/gr00t-tt`, `GR00T_TOKENIZER_DIR=/opt/tt-metal/gr00t_p150/assets/tokenizer/n16`, …), and waits ≤ 1800 s for
+`Application startup complete`. Serves on 20000 or the next free port (printed). tt-cli users after the push:
 `tt serve changh95/GR00T-N1.6-3B-p150` / `tt model stop changh95/GR00T-N1.6-3B-p150`.
 
-Caches on the host (persist across boots): `~/.cache/tt-model/gr00t-n1.6-3b-p150/cache` (JIT kernels, `TT_METAL_CACHE`),
-`~/.cache/tt-model/gr00t-n1.6-3b-p150/weights/gr00t-tt/n16/...` (the port's host plan tier + `.tensorbin` device tier, ~8 GB, written on
-the first boot), weights in `~/.cache/huggingface/hub/models--nvidia--GR00T-N1.6-3B`.
+Caches on the host (persist across boots): `~/.cache/tt-model/gr00t-n1.6-3b-p150/cache` (JIT kernels incl. the megakernel binaries,
+`TT_METAL_CACHE`), `~/.cache/tt-model/gr00t-n1.6-3b-p150/weights/gr00t-tt/n16/...` (the port's host plan tier + `.tensorbin` device tier,
+written on the first boot; the bfp8_b arena is packed from it at every boot), weights in `~/.cache/huggingface/hub/models--nvidia--GR00T-N1.6-3B`.
 
-Measured on the build host with this image (`docs/publish/build-n16.md` has every command and log):
+Measured on the build host with this image (`docs/publish/build-mk-n16.md` of the port project has every command and log):
 
-| boot | wall from `docker run` to READY | notes |
+| boot | wall from `tt-model serve` to READY | notes |
 |---|---:|---|
-| first (cold: empty `/cache` and `/weight-cache`) | 137 s | checkpoint → device-layout plan + `.tensorbin` write, JIT of every kernel of the four traces, trace capture, 3 warm-ups (round-0 build `7e1e1380a5a5`, 2026-09-15 00:30 KST) |
-| second (warm caches, 15 min later) | 13 s | `Model built in 1.2 s` (weight tier in the host page cache), capture 5.2 s (round-0 build) |
-| third (warm caches, 1 h later, the shipped build `feb77048a2ed`) | 21 s | `Model built in 11.2 s` (the 7.9 GB weight tier re-read from disk), capture 4.6 s, 3 warm-ups 60.8 / 59.4 / 59.9 ms; smoke PASS, 50-request bench 56.8 / 59.4 ms, clean stop 2.1 s |
-
-The shipped image (`feb77048a2ed`, fix round 1) differs from the round-0 build only in the server's `GET /info → license` text and in
-`code/` shipping 8 instead of 2,009 `tests/tt/results` files; the tt-metal build, the port, the serve path and the JIT / weight caches are
-identical, so the cold-boot figure was not re-measured. The served latency in the card is the round-1 run of the shipped image.
+| first (cold: empty `/cache` and `/weight-cache`) | 58 s | checkpoint → device-layout plan + `.tensorbin` write, arena pack + upload, JIT of every kernel of the four traces incl. the megakernel, trace capture, 3 warm-ups (this image `0be2c39e4a01`, 2026-09-18 KST); smoke PASS, 50-request bench, clean stop |
+| second (warm caches, minutes later) | 14 s | `Model built in 4.6 s`, capture 3.0 s (JIT cache hits); smoke PASS with bit-identical fidelity, 50-request bench 39.7 / 42.3 ms, clean stop |
+| Stage-1 image `feb77048a2ed` (2026-09-15, for reference) | 137 s cold / 13–21 s warm | the TTNN denoise path of the previous release (`docs/publish/build-n16.md`) |
 
-`push` makes `code/` and `image/` on the Hub exactly the staged trees and replaces `README.md` with the generated card (everything worth
-keeping lives in `card.description` / `card.quickstart`); `tt-model.yaml`, `SERVING.md`, `GPU_COMPARISON.md`, `LICENSE`,
-`LICENSE-NOTICE.md`, `media/` survive (re-uploaded after every push). The front matter (`license: other`, `license_name: nvidia-license`,
-`license_link`, `pipeline_tag: robotics`, `base_model`, extra tags) is restored afterwards with `huggingface_hub.metadata_update`.
+The served latency in the card is the **cold-boot cycle's** 50-request bench of the shipped image (the warm cycle's numbers are the second
+row; both JSONs are kept). `push` makes `code/` and `image/` on the Hub exactly the staged trees and replaces `README.md` with the generated
+card (everything worth keeping lives in `card.description` / `card.quickstart`); `tt-model.yaml`, `SERVING.md`, `GPU_COMPARISON.md`,
+`LICENSE`, `LICENSE-NOTICE.md`, `media/` survive (re-uploaded after every push). The front matter (`license: other`, `license_name:
+nvidia-license`, `license_link`, `pipeline_tag: robotics`, `base_model`, extra tags) is restored afterwards with `huggingface_hub.metadata_update`.
 
 ## 3. Request / response contract
 
 | route | returns |
 |---|---|
 | `GET /health` | `{"status": "ok" \| "starting", "model": "GR00T-N1.6-3B-p150", "device": "blackhole:0"}` — 200 always; `ok` only after warm-up |
-| `GET /info` | model / version / task / hardware / `stage` (`served: stage1-ttnn-traces` + megakernel note) / `weights` (repo, revision, snapshot dir actually loaded) / `source` (port snapshot commit, server version, tt-metal commit, ttnn version) / `inputs` (the full observation contract) / `outputs` / `noise.default_seed` (42) / `limits` / `device_facts` / `warmup_latency_ms` / `warmup_fidelity_vs_golden` / `license` |
+| `GET /info` | model / version / task / hardware / `stage` (`served: stage2-megakernel`, `dit_backend`, `mk_arena_dtype`, `num_command_queues`, `cq1_uploads`, `worker_l1_size` 1395712, `adapter_mem DRAM`, `weights` incl. the skipped ttnn DiT tensors, megakernel head build timings, a note) / `weights` (repo, revision, snapshot dir actually loaded) / `source` (port snapshot commit, server version, tt-metal commit, ttnn version) / `inputs` (the full observation contract) / `outputs` / `noise.default_seed` (42) / `limits` / `device_facts` / `warmup_latency_ms` / `warmup_fidelity_vs_golden` / `license` |
 | `GET /demo` | `{"request": <ready-to-post demo body>, "expected": <golden actions + gates>}` |
 | `GET /v1/models` | OpenAI-shaped stub (`{"object": "list", "data": [{"id": "nvidia/GR00T-N1.6-3B", ...}]}`) so the tt-model ready card does not 404; this is not a chat API |
 | `POST /predict` | one observation → one 16-step action chunk (below) |
@@ -181,7 +199,7 @@ Response (200):
  "normalized": false, "embodiment": "gr1", "embodiment_id": 20, "version": "n16", "model": "GR00T-N1.6-3B-p150", "layout": "gr1",
  "images": {"ego_view_bg_crop_pad_res256_freq20": {"frames": 1, "received_hw": [256, 256]}}, "seq_len": 116, "prompt_tokens": 12,
  "state_dtype": "float32", "noise_source": "seed", "seed": 42,
- "timing_ms": {"decode": 4.0, "encode": 2.8, "device": 57.1, "decode_actions": 0.2, "total": 64.1}}
+ "timing_ms": {"decode": 0.9, "encode": 1.4, "device": 39.6, "decode_actions": 0.2, "total": 42.1}}
 ```
 
 `actions` are physical, un-normalised joint targets (float32 as decimal JSON): the arm and hand groups are the checkpoint's RELATIVE
@@ -195,23 +213,38 @@ synchronous and serialised on one lock; batch is 1.
 
 ## 4. Caveats
 
-* **Stage-1 path.** TTNN ops in four Metal traces (`vision` 11.5 ms, `llm` 9.5, `adapter` 2.0, `denoise` 30.4 — `bench_e2e` medians,
-  `code/models/experimental/gr00t/benchmarks/results/e2e_stage1_n16.json`); DiT matmul weights bfp8_b, everything else bf16. The
-  persistent-megakernel denoise (Stage 2, `tt/megakernel/`, rung K1: 464 GB/s bf16 / 414 GB/s bfp8 weight streaming,
-  `tests/tt/results/mk_k1_summary.md`) is **not** integrated into `Gr00tTT` and not served.
-* **N1.6 fidelity margin.** Final actions reproduce the fp32 golden at PCC 0.99921 on the normalised valid slice (the official bf16 GPU
-  path scores 0.99956 there; boot/smoke gate 0.999). Four intermediate DiT taps miss their per-tap gates on the state-token row 0 only
-  (`dit_out[k=1]` 0.999556 vs 0.99961, `dit_out[k=2]` 0.999496 vs 0.99964, `action_decoder_out[k=1]` 0.999579 vs 0.9997,
-  `action_decoder_out[k=2]` 0.999556 vs 0.99978; 52/56 gated rows pass (57/61 rows incl. the determinism row and the four xfail info rows) —
-  `tests/tt/results/test_e2e_untraced_taps_vs_golden_n16_20260914-232434.json`, `tests/tt/results/d2_summary.md` §3.2). Nothing downstream
-  reads row 0; every action tap passes.
+* **Stage-2 path.** `vision` (10.8 ms), `llm` (9.6) and `adapter` (2.2) are TTNN ops in Metal traces; the `denoise` trace (15.3 ms =
+  3.84 ms per Euler step) is **one persistent megakernel `generic_op`** over the 110 Tensix cores of the p150a: 8 DRAM-reader cores stream
+  the 32-block weight set (0.98 GB bfp8_b per step) once per step to 96 compute cores; the residual stream is row-sharded and the
+  LayerNorm / AdaLN statistics are distributed; attention is a per-head flash SDPA on one core per head (cross blocks with long keys
+  split a head over three cores); gathers and multicasts go through hub cores; the Euler update and the action decoder run inside the
+  kernel (`tests/tt/results/mk_k4_summary.md`, `mk_k4b_summary.md`, `mk_k5_summary.md`, `mk_default_summary.md`). `bench_e2e` medians:
+  `code/models/experimental/gr00t/benchmarks/results/bench_e2e_n16_20260918-050632.json` (the snapshot, 40.67 ms e2e). Against the TTNN
+  denoise on the same tree (`bench_e2e_n16_20260918-032258.json`: 56.20 ms e2e, 30.43 ms denoise) the megakernel saves 27.6 % e2e and
+  49.6 % on the denoise trace — short of the plan's 30 % e2e target (`docs/plan/reviews/WP-K5-review-round1.md`, plan-owner decisions).
+  `GR00T_DIT_BACKEND=ttnn` serves the Stage-1 path on the same image.
+* **What the megakernel costs.** The device is opened with 64 KiB less allocatable L1 per core (`worker_l1_size` 1,395,712 B; the
+  kernel-config ring of 136,192 B holds the compute-core binaries, ≈ 131.7 KB after the K4b size cuts, ≈ 4.5 KB of headroom —
+  `mk_k4b_summary.md` §5), the VL adapter's intermediates live in DRAM (adapter trace 2.24 vs 2.01 ms), and the DiT intermediates
+  (`dit_out[k]`, block taps, AdaLN taps) are **not observable** from the served model: the untraced-taps test produces 23 taps
+  (10 head taps + the backbone taps) instead of the Stage-1 path's 45.
+* **N1.6 fidelity margin.** The served physical actions reproduce the fp32 golden at PCC 0.999232 (all 29 dims; the boot / smoke
+  self-check gate is 0.999 on this quantity), the normalised valid slice at 0.999218 (the official bf16 GPU path scores 0.99956 there). Two intermediate taps miss their per-tap gates on the state-token row 0 only
+  (`action_decoder_out[k=1]` 0.999594 vs 0.9997, `action_decoder_out[k=2]` 0.999598 vs 0.99978; valid-slice PCC 0.999961 / 0.999967);
+  26/28 gated rows pass (29/31 rows incl. the determinism row and the two xfail info rows) —
+  `tests/tt/results/test_e2e_untraced_taps_vs_golden_n16_20260918-045025.json`. The Stage-1 card's `dit_out[k=1,2]` misses are now
+  inside the kernel. Nothing downstream reads row 0; every action tap passes. The megakernel's own gated PCCs on N1.6 are bit-identical to
+  WP-K5's (`mk_default_summary.md` §3.2).
 * **One embodiment, one layout, one camera, batch 1.** `gr1` (GR1 arms + hands + waist), static layout `gr1` (81 image tokens, `L_max` 24,
   `S_pad` 128). `GR00T_LAYOUT=gr1_long_text` (`L_max` 152, `S_pad` 256) exists in the port but was **not** device-validated. Other
   embodiments need their own layout + validation.
 * **Deterministic noise.** Actions are a deterministic function of (observation, seed | noise); the default seed 42 reproduces the
-  reference `initial_noise` bit-exactly (`demo/n16/noise.json: seed_equivalent`). Two identical requests return bit-identical chunks.
-* **First boot is cold**: the checkpoint is converted to the device layout into `/weight-cache` (~8 GB) and every kernel is JIT-compiled
-  into `/cache`; both persist under `~/.cache/tt-model/gr00t-n1.6-3b-p150/`. Measured boot times: §2.
+  reference `initial_noise` bit-exactly (`demo/n16/noise.json: seed_equivalent`). Two identical requests return bit-identical chunks;
+  traced == untraced bit for bit; 5 replays bit-identical; the 2-CQ device equals the 1-CQ device on every probe (`mk_default_summary.md` §4).
+* **First boot is cold**: the checkpoint is converted to the device layout into `/weight-cache` and every kernel (incl. the megakernel)
+  is JIT-compiled into `/cache`; both persist under `~/.cache/tt-model/gr00t-n1.6-3b-p150/`. Measured boot times: §2.
+* **One megakernel model per process.** Building a second megakernel head on the same open device after releasing the first hung the
+  card once on N1.6 (`mk_default_summary.md` §10); the server builds exactly one model per process, as every shipped path does.
 * **Python 3.12 image vs 3.10 host venv.** The image resolves the packages on 3.12 (`requirements.lock`); the host venv is 3.10. The
   code is 3.10/3.12 compatible; the image is the authoritative validation (§2 numbers).
 * **Vendored tokenizer** (`code/gr00t_p150/assets/tokenizer/n16`, NVIDIA License — see `LICENSE-NOTICE.md`): verified bit-exact against the
@@ -220,10 +253,14 @@ synchronous and serialised on one lock; batch is 1.
 
 ## 5. Where the numbers come from
 
-Everything in the card is measured and shipped: `code/models/experimental/gr00t/benchmarks/results/e2e_stage1_n16.json` (Stage-1 latency,
-50 calls, tt-metal `668c2907575`), `code/models/experimental/gr00t/tests/tt/results/test_e2e_{predict_actions_vs_golden,
-traced_equals_untraced,untraced_taps_vs_golden}_n16_20260914-2324*.json` (fidelity, determinism, the four marginal taps),
-`tests/tt/results/d2_summary.md` (device session D2), `tests/tt/results/mk_k1_summary.md` (megakernel K1), `code/gr00t_p150/demo/n16/expected.json`
-(fp32 golden + official bf16 reference actions for the demo step). The RTX 5090 rows are in `GPU_COMPARISON.md`. The served numbers of this
-image (`timing_ms` over 50 requests, cold / warm boot) were measured on the build host as described in §2 and recorded in
-`docs/publish/build-n16.md` of the port project.
+Everything in the card is measured and shipped. Snapshot validation (`commit 009de892c895` in every JSON, run on the frozen tree
+`gr00t-p150-snapshot-mk`, `docs/publish/snapshot-mk.md` of the port project): `code/models/experimental/gr00t/benchmarks/results/
+bench_e2e_n16_20260918-050632.json` (latency, 50 calls after 10 warm-ups, 2 CQs, megakernel / bfp8_b), `code/models/experimental/gr00t/tests/tt/results/
+test_e2e_{predict_actions_vs_golden,traced_equals_untraced,untraced_taps_vs_golden}_n16_20260918-0450*.json` (fidelity, determinism, the two
+marginal taps) and `test_cpu_emulated_e2e_vs_golden_n16_20260918-044939.json` (the CPU emulation of the Stage-1 op path on the device-plan
+weights). Same-tree TTNN reference: `benchmarks/results/bench_e2e_n16_20260918-032258.json`; the Stage-1 release's numbers:
+`benchmarks/results/e2e_stage1_n16.json`. Kernel facts and the default flip: `tests/tt/results/mk_k4_summary.md`, `mk_k4b_summary.md`,
+`mk_k5_summary.md`, `mk_default_summary.md` (+ `mk_tree_20260918-041439.sha256`, the shipped megakernel tree's hashes). Demo golden:
+`code/gr00t_p150/demo/n16/expected.json` (fp32 golden + official bf16 reference actions). The RTX 5090 rows are in `GPU_COMPARISON.md`.
+The served numbers of this image (`timing_ms` over 50 requests, cold / warm boot) were measured on the build host as described in §2 and
+recorded in `docs/publish/build-mk-n16.md` of the port project.

tt-model.yaml

@@ -1,17 +1,18 @@
 # SPDX-License-Identifier: Apache-2.0
-# tt-model-manager container manifest (schema 5.1) for NVIDIA Isaac GR00T N1.6-3B on Blackhole (Stage-1 TTNN trace path).
+# tt-model-manager container manifest (schema 5.1) for NVIDIA Isaac GR00T N1.6-3B on Blackhole
+# (Stage-2: the DiT denoise as one persistent megakernel generic_op; vision / llm / adapter as TTNN traces).
 #
 # Run EVERY tt-model command from this directory: `source.tt_metal` and `extra_code[].root: code` resolve against the
 # process CWD, not against this file. `--out` must lie outside any git checkout (`package` deletes <out>/<name> first).
 # On the build host rootless Docker needs `source /home/deepgadget/experiments/tt-models/bin/docker-env.sh` first.
 #
-#   tt-model package --container tt-model.yaml --out /home/deepgadget/experiments/gr00t/publish/build   # ~2 h cold, runs verify.sh
+#   tt-model package --container tt-model.yaml --out /home/deepgadget/experiments/gr00t/publish/build   # minutes with the ccache mount, ~2 h cold; runs verify.sh
 #   tt-model serve   /home/deepgadget/experiments/gr00t/publish/build/gr00t-n1.6-3b-p150/tt_kernel_manifest.json
-#   python code/gr00t_p150/server/smoke_test.py --url http://127.0.0.1:20000      # PASS/FAIL one-liner
+#   python code/gr00t_p150/server/smoke_test.py --url http://127.0.0.1:20000      # PASS/FAIL one-liner (prints the served stage)
 #   tt-model stop    /home/deepgadget/experiments/gr00t/publish/build/gr00t-n1.6-3b-p150/tt_kernel_manifest.json   # org/name works only after pull/push
 #   tt-model push    /home/deepgadget/experiments/gr00t/publish/build/gr00t-n1.6-3b-p150 --publish
 #
-# The one Blackhole card of the build host is shared: `serve -> smoke -> stop` runs inside ONE bin/with-device.sh hold.
+# The one Blackhole card of the build host is shared: `serve -> smoke -> bench -> stop` runs inside ONE bin/with-device.sh hold.
 schema: "5.1"
 
 repo: changh95/GR00T-N1.6-3B-p150
@@ -56,12 +57,19 @@ source:
     - models/tt_dit/parallel/config.py
     - models/tt_dit/parallel/manager.py
     - models/tt_dit/encoders/qwen3vl/vision_qwen3vl.py          # tt/qwen3vl_vision.py (lazy; N1.7 only)
-    - models/demos/deepseek_v3_b1/unified_kernel_descriptor.py  # tt/megakernel/descriptors.py (lazy; Stage-2 work, not served)
+    - models/demos/deepseek_v3_b1/unified_kernel_descriptor.py  # tt/megakernel/descriptors.py (lazy) -- ON the serve path: the megakernel program descriptor
+    # Device-code headers the megakernel's kernels include from the tree (tt/megakernel/kernels/dit_kernel.cpp
+    # `#include "../../../../../demos/deepseek_v3_b1/unified_kernels/*.hpp"`, JIT-compiled at boot relative to TT_METAL_HOME):
+    # the transitive closure inside models/ is exactly these four; their own includes are tt_metal hw/api headers the image ships.
+    - models/demos/deepseek_v3_b1/unified_kernels/kernel_op_api.hpp
+    - models/demos/deepseek_v3_b1/unified_kernels/kernel_utils.hpp
+    - models/demos/deepseek_v3_b1/unified_kernels/dataflow_utils.hpp
+    - models/demos/deepseek_v3_b1/unified_kernels/mcast.hpp
   # This repo's own code (lands at /opt/tt-metal/<path>; PYTHONPATH=/opt/tt-metal).
   extra_code:
     - root: code
       paths:
-        - models        # code/models/experimental/gr00t: the frozen snapshot of the port (tt-metal commit 5dc0517e, parent 668c2907575)
+        - models        # code/models/experimental/gr00t: the frozen snapshot of the port (tt-metal ref gr00t-p150-snapshot-mk = commit 009de892c895, parent 668c2907575; megakernel tree 7df61c35e9dc7857)
         - gr00t_p150    # FastAPI server, schemas, smoke test, demo observation, vendored Qwen2 tokenizer files (n16)
         - scripts       # download_weights.sh, bench_http.py
   ubuntu: "22.04"
@@ -72,8 +80,8 @@ runtime:
   mesh_shape_env: TT_MESH_SHAPE
   # On top of the kind defaults (fastapi, uvicorn, pydantic>=2, pillow), torch==2.11.0+cpu (auto-pinned from the tree's
   # tt_metal/python_env/requirements-dev.txt) and ttnn's own deps. Every third-party module on the serve path, pinned
-  # where the tree pins it. Resolved on Python 3.12 with uv (dry-run 2026-09-15): numpy 1.26.4, transformers 5.12.1,
-  # opencv-python-headless 4.8.1.78, safetensors 0.8.0, huggingface_hub 1.31.0, loguru 0.7.3, torch 2.11.0+cpu.
+  # where the tree pins it. Resolved on Python 3.12 with uv (image build 2026-09-18, requirements.lock): numpy 1.26.4,
+  # transformers 5.12.1, opencv-python-headless 4.8.1.78, safetensors 0.8.0, huggingface_hub 1.32.0, loguru 0.7.3, torch 2.11.0+cpu.
   # No torchvision: the N1.6 image chain is cv2 + PIL (torchvision is only on the N1.5 path).
   packages:
     - "numpy>=1.24.4,<2"
@@ -91,23 +99,34 @@ serve:
     TT_WEIGHTS_REVISION: "d0814e7ecb19202e7c8468b46098b0b7ef3a6d61"   # == weights.revision == configs.HF_SNAPSHOT_SHAS["n16"]
     TT_METAL_VISIBLE_DEVICES: "0"
     TT_DEVICE_ID: "0"
-    GR00T_VERSION: "n16"                   # the server code is shared with GR00T-N1.5-3B-p150 and never guesses
+    GR00T_VERSION: "n16"                   # the server code is shared with GR00T-N1.5-3B-p150 / N1.7-3B-p150 and never guesses
     GR00T_EMBODIMENT: "gr1"                # GR1 arms + hands + waist, embodiment slot 20; the only device-validated layout
     GR00T_LAYOUT: "gr1"                    # static layout: 1 camera x 81 image tokens, L_max 24 text tokens, S_pad 128
-    GR00T_POLICY: "mixed_dit"              # TTPolicy.dtype_policy: DiT matmul weights bfp8_b, everything else bf16 (validated Stage-1 default)
+    GR00T_POLICY: "mixed_dit"              # TTPolicy.dtype_policy: DiT matmul weights bfp8_b, everything else bf16 (the validated default)
     GR00T_TRACE_LAYOUT: "per_stage"        # four Metal traces: vision / llm / adapter / denoise (the benchmarked layout)
+    GR00T_DIT_BACKEND: "megakernel"        # TTPolicy.dit_backend: the DiT denoise as ONE persistent generic_op (Stage 2, the port's default
+                                           # since 2026-09-18); "ttnn" serves the Stage-1 TTNN op sequence on the same image
+    GR00T_MK_ARENA_DTYPE: "auto"           # TTPolicy.mk_arena_dtype: the megakernel's DRAM weight arena; auto -> bfp8_b for n16 (the validated arena)
+    GR00T_NUM_CQS: "2"                     # command queues of the device (open_model_device): 2 = the port's production path, per-call input
+                                           # writes on CQ 1 overlapping the traces (bit-equal to 1 CQ, tests/tt/test_mk_2cq.py)
     GR00T_TOKENIZER_DIR: "/opt/tt-metal/gr00t_p150/assets/tokenizer/n16"   # vendored Qwen2 tokenizer files (not in the weights repo)
     GR00T_PROJECT_ROOT: "/nonexistent"     # kill the dev-box default the port's prompts.py would otherwise fall back to
     GR00T_GOLDEN_ROOT: "/nonexistent"      # same for common/golden.py (tests only; defensive)
-    GR00T_TT_CACHE: "/weight-cache/gr00t-tt"   # host .pt plan tier + .tensorbin device tier (~8 GB) -> ~/.cache/tt-model/gr00t-n1.6-3b-p150/weights
+    GR00T_TT_CACHE: "/weight-cache/gr00t-tt"   # host .pt plan tier + .tensorbin device tier (~2.6 GB under the megakernel; 7.9 GB on the Stage-1 path) -> ~/.cache/tt-model/gr00t-n1.6-3b-p150/weights
     GR00T_WARMUP_RUNS: "3"                 # traced predict() calls after trace capture, before READY
-    GR00T_WARMUP_PCC_MIN: "0.999"          # boot FAILS when the warm-up actions' PCC vs the shipped fp32 golden is below this
+    GR00T_WARMUP_PCC_MIN: "0.999"          # boot FAILS when the warm-up actions' PCC vs the shipped fp32 golden is below this (served: 0.99923)
 
 # Build-time assertions, run INSIDE the finished image as uid 1000, no device, no weights, no HF_MODEL.
 verify:
   - "import gr00t_p150.server.app as a; assert a.app"
   - "import sys, gr00t_p150.server.app; assert 'ttnn' not in sys.modules and not any(m.startswith('models.experimental.gr00t') for m in sys.modules), 'the server module must import without side effects'"
   - "import models.experimental.gr00t.tt.model as m; assert m.Gr00tTT"
+  - "from models.experimental.gr00t.tt.policy import TTPolicy, DEFAULT_DIT_BACKEND; p = TTPolicy(); assert p.dit_backend == 'megakernel' == DEFAULT_DIT_BACKEND and p.mk_arena_dtype == 'auto' and p.mk_arena_dtype_for('n16') == 'bfp8_b', (p.dit_backend, p.mk_arena_dtype)"
+  - "import models.experimental.gr00t.tt.megakernel.dit_program as d; from models.experimental.gr00t.tt.megakernel import arena, core_map, descriptors; assert d.DiTMegakernel and descriptors.KERNEL_SOURCE.endswith('tt/megakernel/kernels/dit_kernel.cpp')"
+  - "import os; from pathlib import Path; from models.experimental.gr00t.tt.megakernel.descriptors import KERNEL_SOURCE; k = Path(os.environ.get('TT_METAL_HOME', '/opt/tt-metal')) / KERNEL_SOURCE; assert k.is_file() and len(list((k.parent / 'ops').glob('*.hpp'))) == 7, str(k)"
+  - "import os, re; from pathlib import Path; from models.experimental.gr00t.tt.megakernel.descriptors import KERNEL_SOURCE; home = Path(os.environ.get('TT_METAL_HOME', '/opt/tt-metal')); k = home / KERNEL_SOURCE; incs = {m for f in [k, *(k.parent / 'ops').glob('*.hpp')] for m in re.findall(r'#include \"([^\"]*unified_kernels/[^\"]+)\"', f.read_text())}; assert incs, 'no unified_kernels includes found'; missing = [i for i in incs if not (k.parent / i).resolve().is_file() and not (home / i).is_file()]; assert not missing, missing"
+  - "from models.experimental.gr00t.tt.model import open_model_device, device_worker_l1_size, worker_l1_size_for, MK_L1_CUT_BYTES_DEFAULT; from models.experimental.gr00t.tt.policy import TTPolicy; assert MK_L1_CUT_BYTES_DEFAULT == 65536 and worker_l1_size_for(TTPolicy()) is not None and worker_l1_size_for(TTPolicy(dit_backend='ttnn')) is None"
+  - "from gr00t_p150.server.app import STAGE_SERVED, DEFAULT_NUM_CQS, DIT_BACKENDS, MK_ARENA_DTYPES; assert STAGE_SERVED['megakernel'] == 'stage2-megakernel' and STAGE_SERVED['ttnn'] == 'stage1-ttnn-traces' and DEFAULT_NUM_CQS == 2 and set(DIT_BACKENDS) == {'ttnn', 'megakernel'} and 'bfp8_b' in MK_ARENA_DTYPES"
   - "from models.experimental.gr00t.common import configs, preprocessing, normalization, prompts, checkpoint, weights; assert configs.HF_SNAPSHOT_SHAS['n16'] == 'd0814e7ecb19202e7c8468b46098b0b7ef3a6d61' and configs.HF_REPOS['n16'] == 'nvidia/GR00T-N1.6-3B'"
   - "import models.tt_dit.utils.matmul as mm; assert mm.get_matmul_config"
   - "import models.demos.deepseek_v3_b1.unified_kernel_descriptor as u; assert u.UnifiedKernelDescriptor"
@@ -119,12 +138,12 @@ verify:
   - "from gr00t_p150.server.app import load_demo_request; import numpy as np; r = load_demo_request('n16'); assert set(r['images']) == {'ego_view_bg_crop_pad_res256_freq20'} and np.asarray(r['noise']).squeeze().shape == (50, 128)"
   - "from pathlib import Path; assert Path('/opt/tt-metal/models/experimental/gr00t/common/golden_tap_map.json').is_file() and Path('/opt/tt-metal/models/experimental/gr00t/tests/tt/gates/gates_multi.json').is_file()"
   - "from pathlib import Path; assert Path('/opt/tt-metal/gr00t_p150/server/smoke_test.py').is_file() and Path('/opt/tt-metal/scripts/bench_http.py').is_file() and Path('/opt/tt-metal/scripts/download_weights.sh').is_file()"
-  - "import ttnn; assert all(hasattr(ttnn, a) for a in ('begin_trace_capture', 'end_trace_capture', 'execute_trace', 'release_trace', 'copy_host_to_device_tensor', 'MinimalMatmulConfig'))"
+  - "import ttnn; assert all(hasattr(ttnn, a) for a in ('begin_trace_capture', 'end_trace_capture', 'execute_trace', 'release_trace', 'copy_host_to_device_tensor', 'MinimalMatmulConfig', 'generic_op', 'ProgramDescriptor'))"
   - "from gr00t_p150.server.app import parse_mesh_shape as p; assert p('1x1') == p('(1, 1)') == p('1,1') == (1, 1)"
 
 card:
   description: |
-    NVIDIA Isaac GR00T N1.6 (3B vision-language-action policy) running on one Tenstorrent Blackhole p150a via tt-nn: one camera frame, proprioceptive state and a task instruction in, a 16-step chunk of GR1 arm, hand and waist actions out. Stage-1 release: TTNN ops replayed from Metal traces (DiT weights bfp8, everything else bf16); the persistent-megakernel denoise is in progress and not in this image.
+    NVIDIA Isaac GR00T N1.6 (3B vision-language-action policy) running on one Tenstorrent Blackhole p150a via tt-nn: one camera frame, proprioceptive state and a task instruction in, a 16-step chunk of GR1 arm, hand and waist actions out. Stage-2 release: the whole DiT denoise (32 blocks × 4 flow-matching steps) runs as one persistent megakernel streaming bfp8 weights from a DRAM arena; vision, LLM and adapter are TTNN ops (bf16) replayed from Metal traces; `GR00T_DIT_BACKEND=ttnn` selects the Stage-1 denoise.
     Weights: [nvidia/GR00T-N1.6-3B](https://huggingface.co/nvidia/GR00T-N1.6-3B) · Paper: [arXiv:2503.14734](https://arxiv.org/abs/2503.14734) · Project page: [gr00t-n1_6](https://research.nvidia.com/labs/gear/gr00t-n1_6/) · Upstream code: [NVIDIA/Isaac-GR00T](https://github.com/NVIDIA/Isaac-GR00T) (`n1.6.1-release`) · Port: `code/models/experimental/gr00t` and the server `code/gr00t_p150` in this repo
   quickstart: |
     ### Run with tt-cli
@@ -142,18 +161,18 @@ card:
     ```
 
     - `POST /predict`: `images` (`{"ego_view_bg_crop_pad_res256_freq20": <base64 PNG/JPEG>}`), `state` (`{group: [floats]}`: `left_arm` 7, `right_arm` 7, `left_hand` 6, `right_hand` 6, `waist` 3, raw joint values), `instruction` (≤ 24 tokens); optional `seed` (default 42) or `noise` (`[50, 128]`), `return_normalized`.
-    - `GET /health`, `GET /info`, `GET /demo` (the shipped demo request and its golden actions).
+    - `GET /health`, `GET /info` (`stage` names the denoise backend), `GET /demo` (the shipped demo request and its golden actions).
 
     ### Response
 
     ```json
-    {"actions": {"left_arm":   [[-0.0289, 0.1453, 0.0224, -2.1383, -0.0165, -0.0260, -0.0414], "... 16 steps"],
-                 "right_arm":  [[-0.0024, -0.1502, 0.0280, -2.1461, -0.0091, 0.0132, 0.0044], "..."],
-                 "left_hand":  [[0.0097, -0.0177, 0.0041, 0.0078, 0.0539, 0.0281], "..."],
-                 "right_hand": [[-0.2339, -0.2466, -0.2431, -0.2311, -0.2423, 0.3333], "..."],
-                 "waist":      [[-0.0010, -0.0060, 0.0008], "..."]},
+    {"actions": {"left_arm":   [[-0.0289, 0.1453, 0.0224, -2.1383, -0.0165, -0.0225, -0.0428], "... 16 steps"],
+                 "right_arm":  [[-0.0024, -0.1512, 0.0269, -2.1461, -0.0091, 0.0124, 0.0044], "..."],
+                 "left_hand":  [[0.0135, -0.0092, -0.0047, 0.0070, 0.0455, 0.0281], "..."],
+                 "right_hand": [[-0.2253, -0.2421, -0.2345, -0.2311, -0.2338, 0.3385], "..."],
+                 "waist":      [[0.0001, -0.0060, 0.0008], "..."]},
      "action_horizon": 16, "embodiment": "gr1", "seq_len": 116, "seed": 42,
-     "timing_ms": {"decode": 0.8, "encode": 1.5, "device": 56.5, "decode_actions": 0.1, "total": 58.9}}
+     "timing_ms": {"decode": 0.9, "encode": 1.4, "device": 39.6, "decode_actions": 0.2, "total": 42.1}}
     ```
 
     - `actions` are physical, un-normalised joint targets, 16 steps × [7, 7, 6, 6, 3] dims, as `Gr00tPolicy.get_action` returns them for the GR1 data config (arm and hand groups are relative outputs composed onto the request's `state`, `waist` is absolute); `timing_ms.device` covers upload, the trace replays and the readback.
@@ -168,21 +187,23 @@ card:
 
     | Metric | Value |
     |---|---:|
-    | Action chunk vs the fp32 reference (same observation, same initial noise) | PCC `left_arm` 0.99995 · `right_arm` 0.99991 · `right_hand` 0.99312 · `left_hand` / `waist` max\|d\| 0.059 / 0.017 (near-constant groups) · all 29 action dims 0.999209 |
-    | Intermediate taps vs the fp32 reference (45 canonical taps, from pixels) | 52 of 56 gated rows pass; 4 marginal misses on the DiT state-token row that no action reads (`dit_out[k=1,2]` 0.999556 / 0.999496 vs gates 0.99961 / 0.99964, `action_decoder_out[k=1,2]` 0.999579 / 0.999556 vs 0.9997 / 0.99978), caused by bf16 error in the SigLIP2 tower |
-    | Traced vs untraced, determinism | bit-identical; 5 trace replays bit-identical |
-    | Inference on p150a (warm, batch 1, median of 50) | **59.4 ms end-to-end** = host encode 2.0 + upload 3.8 + traces 53.3 (vision 11.5 · LLM 9.5 · adapter 2.0 · denoise 30.4) + readback and decode 0.3 |
-    | Served over HTTP by this image (warm, batch 1) | 56.8 ms device · 59.4 ms server-side incl. PNG decode |
-    | Same policy on an RTX 5090 (same host, NVIDIA's unmodified `Gr00tPolicy`, eager PyTorch bf16 + flash-attn 2) | 63.1 ms end-to-end → p150a 1.06× faster; with `torch.compile` on the DiT the GPU reaches 33.1 ms → GPU 1.8× faster than this Stage-1 path |
+    | Action chunk vs the fp32 reference (same observation, same initial noise) | PCC `left_arm` 0.99995 · `right_arm` 0.99990 · `right_hand` 0.99308 · `left_hand` / `waist` max\|d\| 0.059 / 0.013 (near-constant groups) · all 29 action dims 0.99923 (normalised valid slice 0.99922) |
+    | Intermediate taps vs the fp32 reference (the 23 taps the megakernel path exposes, from pixels; DiT block internals are not observable inside the kernel) | 26 of 28 gated rows pass (29 of 31 rows incl. the determinism row and the two xfail info rows); 2 marginal misses on the DiT state-token row that no action reads (`action_decoder_out[k=1,2]` 0.999594 / 0.999598 vs gates 0.9997 / 0.99978), caused by bf16 error in the SigLIP2 tower; the Stage-1 card's `dit_out[k=1,2]` misses are now inside the kernel |
+    | Megakernel vs the Stage-1 TTNN denoise, same device | `action_pred_normalized` PCC ≥ 0.999994 on every N1.6 reference sample |
+    | Traced vs untraced, determinism | bit-identical (13 of 13 rows); 5 trace replays bit-identical; 2 CQs == 1 CQ |
+    | Inference on p150a (warm, batch 1, median of 50) | **40.7 ms end-to-end** (p90 40.8) = host encode 1.4 + upload 1.4 + traces 37.8 (vision 10.8 · LLM 9.6 · adapter 2.2 · denoise 15.3 = 3.84 ms per step) + readback and decode 0.1 |
+    | Served over HTTP by this image (warm, batch 1, 50 requests) | 39.6 ms device · 42.1 ms server-side incl. PNG decode (p90 42.5) |
+    | Same policy on an RTX 5090 (same host, NVIDIA's unmodified `Gr00tPolicy`, eager PyTorch bf16 + flash-attn 2) | 63.1 ms end-to-end → p150a 1.55× faster; with `torch.compile` on the DiT the GPU reaches 33.1 ms → GPU 1.23× faster than this Stage-2 path |
 
     ### Caveats
 
-    - Stage-1 path: TTNN ops in four Metal traces, DiT matmul weights bfp8, everything else bf16. The megakernel denoise (weight streaming measured at 464 GB/s bf16 / 414 GB/s bfp8 over the 32-block N1.6 weight set) is not in this image.
-    - Fidelity margin: the served chunk matches the fp32 reference at PCC 0.99921 on the normalised valid slice (NVIDIA's own bf16 GPU run scores 0.99956), 2e-4 above the 0.999 boot self-check.
+    - Stage-2 path: the `denoise` trace is one persistent `generic_op` on all 110 Tensix cores (8 readers stream the 32-block weight set once per step from a 0.98 GB bfp8 DRAM arena to 96 compute cores; distributed LayerNorm, per-head flash SDPA, Euler update and decoder inside the kernel). Against the ttnn denoise on the same tree (e2e 56.2 ms, denoise 30.4) it saves 27.6 % / 49.6 %, short of the planned 30 % e2e; `GR00T_DIT_BACKEND=ttnn` selects that path.
+    - Fidelity margin: the served chunk matches the fp32 reference at PCC 0.99923 (physical actions; 0.99922 on the normalised valid slice, where NVIDIA's own bf16 GPU run scores 0.99956), 2.3e-4 above the 0.999 boot self-check.
+    - Costs of the megakernel: 64 KiB less allocatable L1 per core (the kernel-config ring is 136,192 B, the compute-core binaries take ≈ 131.7 KB of it), the VL adapter's intermediates in DRAM (+0.2 ms adapter trace), and no DiT-internal taps (10 head taps + the backbone taps remain gated).
     - One embodiment and layout: GR1 (embodiment slot 20), one `ego_view_bg_crop_pad_res256_freq20` camera, instruction ≤ 24 tokens; anything else is refused with 400. Batch 1.
     - Noise is seeded (seed 42, as the deployed policy); pass `seed` or `noise` for another draw. Outputs are the base checkpoint's actions for the GR1 sim data config.
     - Not an OpenAI-compatible API; `GET /v1/models` is a stub.
-    - First boot is cold (weight conversion + kernel JIT, 137 s); later boots take 13–21 s. Validated on tt-metal `668c2907575`, single p150a only. Details: [`SERVING.md`](SERVING.md), [`GPU_COMPARISON.md`](GPU_COMPARISON.md).
+    - First boot is cold (weight conversion + kernel JIT incl. the megakernel, 58 s); later boots take 14 s. Validated on tt-metal `668c2907575`, single p150a only. Details: [`SERVING.md`](SERVING.md), [`GPU_COMPARISON.md`](GPU_COMPARISON.md).
 
     ### Licensing