# Serving GR00T N1.6-3B (`nvidia/GR00T-N1.6-3B`) on Blackhole with tt-model-manager

This repo is a **tt-model container package source**: `tt-model.yaml` + the code under `code/`. `tt-model package
--container` builds the OCI image (tt-metal from source + the port + the HTTP stack), `tt-model serve` runs it against
the card, `tt-model push` publishes it here. The server is `code/gr00t_p150/server/app.py` (FastAPI, `kind:
tt-dit-server`); the device model is the tt-nn port `code/models/experimental/gr00t` (Stage-2 path: vision / LLM / adapter
as TTNN ops captured in three Metal traces, the whole DiT denoise as **one persistent megakernel `generic_op`** in the
fourth trace, streaming its weights from a bfp8_b DRAM arena; everything else bf16). Weights are a pinned pointer, never in the image.

| | |
|---|---|
| tt-metal | main `668c290757550588d0ce46b180c344a462a2aaf5` (`v0.79.0-dev20260914-1-g668c290757`), built from source inside the image; torch pin 2.11.0 (+cpu in the image) |
| weights | `nvidia/GR00T-N1.6-3B` @ `d0814e7ecb19202e7c8468b46098b0b7ef3a6d61` — `model-0000{1,2}-of-00002.safetensors` (6,573,217,664 B of bf16 tensors; 6,573,374,552 B on disk in 2 files), `model.safetensors.index.json`, `config.json`, `processor_config.json`, `statistics.json`, `embodiment_id.json`, `LICENSE` (NVIDIA License, non-commercial) |
| app | `gr00t_p150.server.app:app` (uvicorn `--lifespan on`; the lifespan resolves the snapshot, resolves the `TTPolicy`, opens the device for it, builds `Gr00tTT`, captures the traces, runs `GR00T_WARMUP_RUNS` traced predictions and checks them against the shipped fp32 golden before READY) |
| device recipe | `models.experimental.gr00t.tt.model.open_model_device(policy, device_id, trace_region_size=64 MiB, l1_small_size=32768, num_command_queues=2)`: for `dit_backend=megakernel` the device is opened with `worker_l1_size` 1,461,248 − 65,536 = **1,395,712 B** (a 136,192 B kernel-config ring for the megakernel binaries), program cache on; asserts the 11×10 compute / 8×1 DRAM grid of a p150. `GR00T_DIT_BACKEND=ttnn` opens the firmware-default L1 |
| port source | `code/models/experimental/gr00t` = the frozen snapshot commit `009de892c895917ad6abe9a6c36518f00316ed4b` (ref `gr00t-p150-snapshot-mk`, parent = `668c2907575`) of the tt-metal working tree `models/experimental/gr00t`; megakernel tree `tt/megakernel/**` = the K4b-final checkpoint (`tests/tt/results/mk_tree_20260918-041439.sha256`, digest `7df61c35e9dc7857`); `code/gr00t_p150` (server) and `code/scripts` were written for this release |
| only served configuration | version `n16`, embodiment `gr1` (slot 20), static layout `gr1` (1 camera × 81 image tokens, `L_max` 24 text tokens, LLM sequence padded to 128), `mixed_dit` dtype policy, `per_stage` trace layout, `dit_backend` **`megakernel`** with the `bfp8_b` arena, 2 command queues, batch 1 |

Below, `$ROOT` = the tt-models tooling checkout (`/home/deepgadget/experiments/tt-models` on the build host: `tt-model` CLI in
`$ROOT/.venv/bin`, rootless Docker env in `$ROOT/bin/docker-env.sh`), `$TREE` = the tt-metal tree with the built `python_env`
(`/home/deepgadget/experiments/gr00t/tt-metal`), `$REPO` = this directory.

## Layout

```
tt-model.yaml                 authoring manifest (schema 5.1) -- the whole build/serve interface
tt_kernel_manifest.json       GENERATED wire manifest: weights pointer, image tag/digest, runtime, serve, verify, built provenance
requirements.lock             GENERATED `pip freeze` of the image venv (Python 3.12, torch 2.11.0+cpu, ttnn <scm>)
image/                        GENERATED OCI layout of the image (~3 GB)
README.md                     the card (generated by `tt-model package` from card.description / card.quickstart; front matter restored after push)
SERVING.md, GPU_COMPARISON.md, LICENSE, LICENSE-NOTICE.md     authored
media/demo_ego_view.png       the GR1 PickNPlace demo frame (256x256), media/demo_observation.json the matching raw state + instruction
media/demo_actions_vs_golden.png   served action chunk vs the fp32 reference, one panel per action dimension
code/models/experimental/gr00t/    the port: common/ (host preprocessing, prompts, normalisation, checkpoint, weight plan),
                                   reference/ (fp32 torch reference), tt/ (device model, layers, traces), tt/megakernel/ (the DiT
                                   megakernel: arena, core map, descriptors, dit_program.py, kernels/*.cpp|hpp), tests/ (CPU + device
                                   suites; tests/tt/results/ = the summaries + the snapshot-validation JSONs), benchmarks/ (bench_e2e etc. + results)
code/models/common/utility_functions.py, code/models/tt_dit/{utils,layers,parallel,encoders/qwen3vl}/..., code/models/demos/deepseek_v3_b1/{unified_kernel_descriptor.py, unified_kernels/{kernel_op_api,kernel_utils,dataflow_utils,mcast}.hpp}
                                   the tt-metal files the port imports from the tree other than itself (staged by tt-model from `source.code`; the four
                                   .hpp are device-code headers dit_kernel.cpp includes, JIT-compiled at boot)
code/gr00t_p150/              server/{app.py, schemas.py, smoke_test.py}, demo/n16/ (canonical demo observation + golden), assets/tokenizer/n16/ (vendored Qwen2 tokenizer)
code/scripts/                 download_weights.sh (pinned snapshot -> HF cache), bench_http.py (HTTP latency, 50 requests)
```

In the image the code lands at `/opt/tt-metal/<path>` with `PYTHONPATH=/opt/tt-metal` and `TT_METAL_HOME=/opt/tt-metal`, which is
exactly the tree layout the port was written for (`models.experimental.gr00t.*`, `models.tt_dit.*` absolute imports; `models` is a
namespace package; the megakernel source `models/experimental/gr00t/tt/megakernel/kernels/dit_kernel.cpp` is JIT-compiled by tt-metal
relative to `TT_METAL_HOME`).

## 1. Run on the HOST (hardware validation, no Docker)

The tree venv `$TREE/python_env` (Python 3.10, torch 2.11.0+cpu, ttnn editable, transformers 5.12.1, opencv 4.8.1) has everything
except `fastapi` / `uvicorn`. Do not install into the tree venv; put the HTTP stack in a side directory and prepend it to `PYTHONPATH`.
`$REPO/code` must be FIRST on `PYTHONPATH` so `models.experimental.gr00t` resolves to the shipped snapshot (the tree's `models/` is still
needed for `models.tt_dit.*` / `models.demos.*` on the host; in the image the staged `code/` is the only `models` tree).

```bash
export TREE=/home/deepgadget/experiments/gr00t/tt-metal
export PATH=$HOME/.local/bin:$PATH                                       # uv
HTTP=/home/deepgadget/experiments/gr00t/publish/http-side                # any writable dir outside the trees
uv pip install --python $TREE/python_env/bin/python --target $HTTP fastapi uvicorn "pydantic>=2"

cd $REPO
export PYTHONPATH=$PWD/code:$TREE:$TREE/ttnn:$TREE/tools:$HTTP
export TT_METAL_HOME=$TREE ARCH_NAME=blackhole
export TT_METAL_CACHE=$HOME/.cache/tt-metal-cache-gr00t                  # JIT kernels (incl. the megakernel binaries)
export HF_MODEL=nvidia/GR00T-N1.6-3B
export TT_WEIGHTS_REVISION=d0814e7ecb19202e7c8468b46098b0b7ef3a6d61      # scripts/download_weights.sh puts it in ~/.cache/huggingface
export TT_MESH_SHAPE=1x1 TT_DEVICE_ID=0 TT_METAL_VISIBLE_DEVICES=0
export GR00T_VERSION=n16 GR00T_EMBODIMENT=gr1 GR00T_LAYOUT=gr1 GR00T_POLICY=mixed_dit GR00T_TRACE_LAYOUT=per_stage
export GR00T_DIT_BACKEND=megakernel GR00T_MK_ARENA_DTYPE=auto GR00T_NUM_CQS=2   # the image's serve.env; unset = the same port defaults
export GR00T_TOKENIZER_DIR=$PWD/code/gr00t_p150/assets/tokenizer/n16
export GR00T_TT_CACHE=$HOME/.cache/gr00t-tt                              # host .pt plan tier + .tensorbin device tier (~2.6 GB under the megakernel for n16; 7.9 GB on the Stage-1 path)
export GR00T_WARMUP_RUNS=3

# import check, no device (what the image's verify.sh does):
$TREE/python_env/bin/python -c "import gr00t_p150.server.app as a; assert a.app; print('ok')"

# serve (opens the device for the policy, loads the weights, builds the megakernel arena, captures the four traces,
# 3 warm-up predictions checked against the golden, then READY). The one Blackhole card of the build host is shared
# with other agents: run server + client + shutdown inside ONE lock hold.
DEVICE_LOCK_TIMEOUT=14400 /home/deepgadget/experiments/gr00t/bin/with-device.sh bash -c '
  $TREE/python_env/bin/python -m uvicorn --host 127.0.0.1 --port 20016 --lifespan on gr00t_p150.server.app:app & UV=$!
  python3 code/gr00t_p150/server/smoke_test.py --url http://127.0.0.1:20016 --wait 1800 --out /tmp/gr00t-n16-smoke.json; RC=$?
  python3 code/scripts/bench_http.py --url http://127.0.0.1:20016 --n 50 --out /tmp/gr00t-n16-bench.json
  kill -TERM $UV; wait $UV; exit $RC'
```

Boot log landmarks (they drive `tt-model serve`'s boot checklist): `Loading weights: nvidia/GR00T-N1.6-3B @ d0814e7e…` →
`Tokenizer files: …` → `Opening device 0 (trace_region_size=67108864, l1_small_size=32768, num_command_queues=2, dit_backend=megakernel,
mk_arena_dtype=bfp8_b)` → `Loading pipeline: Gr00tTT.from_pretrained(...)` → `Model built in N s (weights …, 503 tensors, 2744 MB on device,
cache path warm|cold; dit_backend megakernel, arena bfp8_b, adapter_mem DRAM, …; ttnn DiT weights skipped 534 tensors / 1032 MB)` →
`Warming up: capture per_stage traces on the n16 demo observation, then 3 traced predict(s)` → `Warmup k/3: … ms` →
`Warm-up fidelity vs golden fp32: actions PCC 0.99923…` → `Warmup complete (…)` → uvicorn `Application startup complete`. Startup
failures raise and uvicorn exits non-zero (no CPU fallback); a warm-up PCC below `GR00T_WARMUP_PCC_MIN` (0.999) fails the boot. Stop with
SIGTERM / Ctrl-C: the lifespan releases the traces and device tensors and closes the device (`Releasing traces and device tensors`,
`Closing device`).

Expected smoke line on the demo observation (host, live tree at the snapshot state, **cold** `TT_METAL_CACHE`, warm weight tiers;
`publish/logs/smoke_n16_20260918-043318.log` of the build host, `tests/tt/results/mk_default_summary.md` §7):

```
PASS GR00T-N1.6-3B-p150 n16: PCC(actions)=0.999232 max|d|=0.0925 PCC(action_pred_valid)=0.999218 repeat_maxdiff=0.0e+00 seed_path_ok device_ms=40.21/39.37 total_ms=47.94/44.63 wall_ms=55 per_group[left_arm=0.99995/0.0176 right_arm=0.99990/0.0423 left_hand=0.79648/0.0589 right_hand=0.99308/0.0925 waist=0.78038/0.0134]
```

That host boot (every kernel JIT-compiled, incl. the megakernel): model built 4.3 s, `warm_and_capture` 33.1 s, 5 warm-ups 41.4 ms, READY
after 41.2 s (`/info -> warmup_latency_ms`). Offline overrides: `GR00T_WEIGHTS_DIR=<dir with the seven snapshot files>` (shimmed into a
private hub cache so the port's `configs.snapshot_dir("n16")` resolves to it); `GR00T_TOKENIZER_DIR=<dir>`. Every variable is read in
the lifespan, never at import.

Environment knobs of the Stage-2 server (all read in the lifespan; the image's `serve.env` sets the first three explicitly):

| variable | image value | meaning |
|---|---|---|
| `GR00T_DIT_BACKEND` | `megakernel` | `TTPolicy.dit_backend`: `megakernel` (the DiT denoise as one persistent `generic_op`, Stage 2, the port's default) or `ttnn` (the Stage-1 TTNN op sequence, for A/B on the same image). Reported in `/info -> stage.served` (`stage2-megakernel` / `stage1-ttnn-traces`) |
| `GR00T_MK_ARENA_DTYPE` | `auto` (-> `bfp8_b`) | `TTPolicy.mk_arena_dtype` of the megakernel's DRAM weight arena: `auto` / `bf16` / `bfp8_b`; the resolved value is `/info -> stage.mk_arena_dtype` |
| `GR00T_NUM_CQS` | `2` | command queues the device is opened with: `2` = per-call input writes on CQ 1 overlapping the traces (bit-equal to `1` on every probe, `tests/tt/results/mk_default_summary.md` §4) |
| `GR00T_WARMUP_PCC_MIN` | `0.999` | boot self-check on the warm-up chunk vs the shipped fp32 golden; the served N1.6 chunk scores 0.99923 |

## 2. Package, serve, push (Docker)

Rootless Docker on the build host needs `source $ROOT/bin/docker-env.sh` first (PATH + `DOCKER_HOST`; the bare `docker` is podman).
**Run every `tt-model` command from this directory**: `source.tt_metal` and `extra_code[].root: code` resolve against the process CWD.
`--out` points outside any git checkout because `stage()` deletes `<out>/gr00t-n1.6-3b-p150` before rebuilding.

```bash
source $ROOT/bin/docker-env.sh
cd $REPO
# offline validation (must print VALID)
$ROOT/.venv/bin/python -c "from tt_kernel.container_manifest import load_container_manifest as L; m = L('tt-model.yaml', check_sources=True); print('VALID', m.name, m.weights_ref)"

$ROOT/.venv/bin/tt-model package --container tt-model.yaml --out /home/deepgadget/experiments/gr00t/publish/build   # minutes with the ccache mount (~2 h cold), runs verify.sh
# serve -> smoke -> bench -> stop inside ONE device-lock hold (the container keeps the card after `serve` returns)
DEVICE_LOCK_TIMEOUT=14400 /home/deepgadget/experiments/gr00t/bin/with-device.sh bash -c '
  source $ROOT/bin/docker-env.sh
  $ROOT/.venv/bin/tt-model serve /home/deepgadget/experiments/gr00t/publish/build/gr00t-n1.6-3b-p150/tt_kernel_manifest.json || exit 1
  python3 code/gr00t_p150/server/smoke_test.py --url http://127.0.0.1:20000 --out /tmp/gr00t-n16-smoke.json
  python3 code/scripts/bench_http.py --url http://127.0.0.1:20000 --n 50 --out /tmp/gr00t-n16-bench.json
  $ROOT/.venv/bin/tt-model stop /home/deepgadget/experiments/gr00t/publish/build/gr00t-n1.6-3b-p150/tt_kernel_manifest.json'   # org/name is a valid target only after pull/push
$ROOT/.venv/bin/tt-model push /home/deepgadget/experiments/gr00t/publish/build/gr00t-n1.6-3b-p150 --publish
```

`serve` pre-downloads the pinned files into `~/.cache/huggingface` (a metadata no-op when present), then
`docker run --user 0:0 --device /dev/tenstorrent --ipc host --mount /dev/hugepages-1G -v ~/.cache/huggingface:/hf
-v ~/.cache/tt-model/gr00t-n1.6-3b-p150/{cache,weights,tensors}:/{cache,weight-cache,tensor-cache} -p 20000:20000` with exactly
`HF_MODEL=nvidia/GR00T-N1.6-3B`, `MESH_DEVICE=P150`, `TT_MESH_SHAPE=1x1`, `HF_HOME=/hf`, `TT_METAL_CACHE=/cache`,
`TT_DIT_CACHE_DIR=/weight-cache`, `TT_CACHE_PATH=/tensor-cache` and the `serve.env` block of `tt-model.yaml`
(`TT_WEIGHTS_REVISION`, `GR00T_VERSION=n16`, `GR00T_DIT_BACKEND=megakernel`, `GR00T_MK_ARENA_DTYPE=auto`, `GR00T_NUM_CQS=2`,
`GR00T_TT_CACHE=/weight-cache/gr00t-tt`, `GR00T_TOKENIZER_DIR=/opt/tt-metal/gr00t_p150/assets/tokenizer/n16`, …), and waits ≤ 1800 s for
`Application startup complete`. Serves on 20000 or the next free port (printed). tt-cli users after the push:
`tt serve changh95/GR00T-N1.6-3B-p150` / `tt model stop changh95/GR00T-N1.6-3B-p150`.

Caches on the host (persist across boots): `~/.cache/tt-model/gr00t-n1.6-3b-p150/cache` (JIT kernels incl. the megakernel binaries,
`TT_METAL_CACHE`), `~/.cache/tt-model/gr00t-n1.6-3b-p150/weights/gr00t-tt/n16/...` (the port's host plan tier + `.tensorbin` device tier,
written on the first boot; the bfp8_b arena is packed from it at every boot), weights in `~/.cache/huggingface/hub/models--nvidia--GR00T-N1.6-3B`.

Measured on the build host with this image (`docs/publish/build-mk-n16.md` of the port project has every command and log):

| boot | wall from `tt-model serve` to READY | notes |
|---|---:|---|
| first (cold: empty `/cache` and `/weight-cache`) | 58 s | checkpoint → device-layout plan + `.tensorbin` write, arena pack + upload, JIT of every kernel of the four traces incl. the megakernel, trace capture, 3 warm-ups (this image `0be2c39e4a01`, 2026-09-18 KST); smoke PASS, 50-request bench, clean stop |
| second (warm caches, minutes later) | 14 s | `Model built in 4.6 s`, capture 3.0 s (JIT cache hits); smoke PASS with bit-identical fidelity, 50-request bench 39.7 / 42.3 ms, clean stop |
| Stage-1 image `feb77048a2ed` (2026-09-15, for reference) | 137 s cold / 13–21 s warm | the TTNN denoise path of the previous release (`docs/publish/build-n16.md`) |

The served latency in the card is the **cold-boot cycle's** 50-request bench of the shipped image (the warm cycle's numbers are the second
row; both JSONs are kept). `push` makes `code/` and `image/` on the Hub exactly the staged trees and replaces `README.md` with the generated
card (everything worth keeping lives in `card.description` / `card.quickstart`); `tt-model.yaml`, `SERVING.md`, `GPU_COMPARISON.md`,
`LICENSE`, `LICENSE-NOTICE.md`, `media/` survive (re-uploaded after every push). The front matter (`license: other`, `license_name:
nvidia-license`, `license_link`, `pipeline_tag: robotics`, `base_model`, extra tags) is restored afterwards with `huggingface_hub.metadata_update`.

## 3. Request / response contract

| route | returns |
|---|---|
| `GET /health` | `{"status": "ok" \| "starting", "model": "GR00T-N1.6-3B-p150", "device": "blackhole:0"}` — 200 always; `ok` only after warm-up |
| `GET /info` | model / version / task / hardware / `stage` (`served: stage2-megakernel`, `dit_backend`, `mk_arena_dtype`, `num_command_queues`, `cq1_uploads`, `worker_l1_size` 1395712, `adapter_mem DRAM`, `weights` incl. the skipped ttnn DiT tensors, megakernel head build timings, a note) / `weights` (repo, revision, snapshot dir actually loaded) / `source` (port snapshot commit, server version, tt-metal commit, ttnn version) / `inputs` (the full observation contract) / `outputs` / `noise.default_seed` (42) / `limits` / `device_facts` / `warmup_latency_ms` / `warmup_fidelity_vs_golden` / `license` |
| `GET /demo` | `{"request": <ready-to-post demo body>, "expected": <golden actions + gates>}` |
| `GET /v1/models` | OpenAI-shaped stub (`{"object": "list", "data": [{"id": "nvidia/GR00T-N1.6-3B", ...}]}`) so the tt-model ready card does not 404; this is not a chat API |
| `POST /predict` | one observation → one 16-step action chunk (below) |

`POST /predict` request (JSON; unknown fields → 422):

| field | type | meaning |
|---|---|---|
| `images` | `{camera key: base64 PNG/JPEG}` or `{camera key: [one frame]}` | exactly one camera, `ego_view_bg_crop_pad_res256_freq20`; any resolution with sides in [64, 4096]; the frame goes through the version's own eval chain (letterbox → 256 INTER_AREA → 0.95 centre crop → 256 → PIL bicubic 252 → Eagle3 normalisation), so no client-controlled shape reaches ttnn |
| `state` | `{group: [D floats]}` (or `[[D floats]]`) | raw physical joint state: `left_arm` 7, `right_arm` 7, `left_hand` 6, `right_hand` 6, `waist` 3; finite |
| `instruction` | str (aliases `language`, `prompt`) | task text, 1–2000 chars; ≤ 24 BPE tokens in this layout (longer → 400: `n_text exceeds L_max`); leading/trailing whitespace stripped |
| `embodiment` | str, optional | must be `gr1` |
| `seed` | int ≥ 0, optional | CPU-generator seed of the `[1, 50, 128]` flow-matching noise; default 42 (the deployed policy's seed) |
| `noise` | `[50, 128]` or `[1, 50, 128]` floats, optional | explicit initial noise (the smoke test sends the reference noise); exclusive with `seed` |
| `state_dtype` | `float32` \| `float64`, optional | default `float32` for N1.6 (the reference dataset's dtype) |
| `return_normalized` | bool, optional | also return `action_pred_normalized` (`[50, 128]` model-space chunk) and `action_pred_valid_hd` (`[16, 29]`) |

Response (200):

```json
{"actions": {"left_arm": [[7 floats] x 16], "right_arm": [...], "left_hand": [[6] x 16], "right_hand": [...], "waist": [[3] x 16]},
 "action_horizon": 16, "action_keys": ["left_arm", "right_arm", "left_hand", "right_hand", "waist"], "action_dims": {"left_arm": 7, "right_arm": 7, "left_hand": 6, "right_hand": 6, "waist": 3},
 "normalized": false, "embodiment": "gr1", "embodiment_id": 20, "version": "n16", "model": "GR00T-N1.6-3B-p150", "layout": "gr1",
 "images": {"ego_view_bg_crop_pad_res256_freq20": {"frames": 1, "received_hw": [256, 256]}}, "seq_len": 116, "prompt_tokens": 12,
 "state_dtype": "float32", "noise_source": "seed", "seed": 42,
 "timing_ms": {"decode": 0.9, "encode": 1.4, "device": 39.6, "decode_actions": 0.2, "total": 42.1}}
```

`actions` are physical, un-normalised joint targets (float32 as decimal JSON): the arm and hand groups are the checkpoint's RELATIVE
outputs already composed onto the request's `state` (per-step min/max un-normalisation from `statistics.json` + the raw last state,
clipped), `waist` is ABSOLUTE — exactly `Gr00tPolicy.get_action` for the GR1 data config. `timing_ms`: `decode` = base64 + PNG decode +
validation, `encode` = the port's host preprocessing, `device` = `predict_normalized` (input writes, `execute_trace` × 4, one blocking
readback), `decode_actions` = un-normalisation, `total` = handler wall time. Errors: **400** (wrong/missing camera or state group, wrong
state width, non-finite state, undecodable image, frame outside [64, 4096], empty/too-long instruction, other embodiment, `seed` +
`noise` together, malformed noise), **422** (schema), **503** while starting, **500** `Type: text` on a device failure. Handlers are
synchronous and serialised on one lock; batch is 1.

## 4. Caveats

* **Stage-2 path.** `vision` (10.8 ms), `llm` (9.6) and `adapter` (2.2) are TTNN ops in Metal traces; the `denoise` trace (15.3 ms =
  3.84 ms per Euler step) is **one persistent megakernel `generic_op`** over the 110 Tensix cores of the p150a: 8 DRAM-reader cores stream
  the 32-block weight set (0.98 GB bfp8_b per step) once per step to 96 compute cores; the residual stream is row-sharded and the
  LayerNorm / AdaLN statistics are distributed; attention is a per-head flash SDPA on one core per head (cross blocks with long keys
  split a head over three cores); gathers and multicasts go through hub cores; the Euler update and the action decoder run inside the
  kernel (`tests/tt/results/mk_k4_summary.md`, `mk_k4b_summary.md`, `mk_k5_summary.md`, `mk_default_summary.md`). `bench_e2e` medians:
  `code/models/experimental/gr00t/benchmarks/results/bench_e2e_n16_20260918-050632.json` (the snapshot, 40.67 ms e2e). Against the TTNN
  denoise on the same tree (`bench_e2e_n16_20260918-032258.json`: 56.20 ms e2e, 30.43 ms denoise) the megakernel saves 27.6 % e2e and
  49.6 % on the denoise trace — short of the plan's 30 % e2e target (`docs/plan/reviews/WP-K5-review-round1.md`, plan-owner decisions).
  `GR00T_DIT_BACKEND=ttnn` serves the Stage-1 path on the same image.
* **What the megakernel costs.** The device is opened with 64 KiB less allocatable L1 per core (`worker_l1_size` 1,395,712 B; the
  kernel-config ring of 136,192 B holds the compute-core binaries, ≈ 131.7 KB after the K4b size cuts, ≈ 4.5 KB of headroom —
  `mk_k4b_summary.md` §5), the VL adapter's intermediates live in DRAM (adapter trace 2.24 vs 2.01 ms), and the DiT intermediates
  (`dit_out[k]`, block taps, AdaLN taps) are **not observable** from the served model: the untraced-taps test produces 23 taps
  (10 head taps + the backbone taps) instead of the Stage-1 path's 45.
* **N1.6 fidelity margin.** The served physical actions reproduce the fp32 golden at PCC 0.999232 (all 29 dims; the boot / smoke
  self-check gate is 0.999 on this quantity), the normalised valid slice at 0.999218 (the official bf16 GPU path scores 0.99956 there). Two intermediate taps miss their per-tap gates on the state-token row 0 only
  (`action_decoder_out[k=1]` 0.999594 vs 0.9997, `action_decoder_out[k=2]` 0.999598 vs 0.99978; valid-slice PCC 0.999961 / 0.999967);
  26/28 gated rows pass (29/31 rows incl. the determinism row and the two xfail info rows) —
  `tests/tt/results/test_e2e_untraced_taps_vs_golden_n16_20260918-045025.json`. The Stage-1 card's `dit_out[k=1,2]` misses are now
  inside the kernel. Nothing downstream reads row 0; every action tap passes. The megakernel's own gated PCCs on N1.6 are bit-identical to
  WP-K5's (`mk_default_summary.md` §3.2).
* **One embodiment, one layout, one camera, batch 1.** `gr1` (GR1 arms + hands + waist), static layout `gr1` (81 image tokens, `L_max` 24,
  `S_pad` 128). `GR00T_LAYOUT=gr1_long_text` (`L_max` 152, `S_pad` 256) exists in the port but was **not** device-validated. Other
  embodiments need their own layout + validation.
* **Deterministic noise.** Actions are a deterministic function of (observation, seed | noise); the default seed 42 reproduces the
  reference `initial_noise` bit-exactly (`demo/n16/noise.json: seed_equivalent`). Two identical requests return bit-identical chunks;
  traced == untraced bit for bit; 5 replays bit-identical; the 2-CQ device equals the 1-CQ device on every probe (`mk_default_summary.md` §4).
* **First boot is cold**: the checkpoint is converted to the device layout into `/weight-cache` and every kernel (incl. the megakernel)
  is JIT-compiled into `/cache`; both persist under `~/.cache/tt-model/gr00t-n1.6-3b-p150/`. Measured boot times: §2.
* **One megakernel model per process.** Building a second megakernel head on the same open device after releasing the first hung the
  card once on N1.6 (`mk_default_summary.md` §10); the server builds exactly one model per process, as every shipped path does.
* **Python 3.12 image vs 3.10 host venv.** The image resolves the packages on 3.12 (`requirements.lock`); the host venv is 3.10. The
  code is 3.10/3.12 compatible; the image is the authoritative validation (§2 numbers).
* **Vendored tokenizer** (`code/gr00t_p150/assets/tokenizer/n16`, NVIDIA License — see `LICENSE-NOTICE.md`): verified bit-exact against the
  Isaac-GR00T checkout and the golden `input_ids`.
* `tt-model curl` / `GET /v1/models` are OpenAI-shaped and are not this API; use the routes above.

## 5. Where the numbers come from

Everything in the card is measured and shipped. Snapshot validation (`commit 009de892c895` in every JSON, run on the frozen tree
`gr00t-p150-snapshot-mk`, `docs/publish/snapshot-mk.md` of the port project): `code/models/experimental/gr00t/benchmarks/results/
bench_e2e_n16_20260918-050632.json` (latency, 50 calls after 10 warm-ups, 2 CQs, megakernel / bfp8_b), `code/models/experimental/gr00t/tests/tt/results/
test_e2e_{predict_actions_vs_golden,traced_equals_untraced,untraced_taps_vs_golden}_n16_20260918-0450*.json` (fidelity, determinism, the two
marginal taps) and `test_cpu_emulated_e2e_vs_golden_n16_20260918-044939.json` (the CPU emulation of the Stage-1 op path on the device-plan
weights). Same-tree TTNN reference: `benchmarks/results/bench_e2e_n16_20260918-032258.json`; the Stage-1 release's numbers:
`benchmarks/results/e2e_stage1_n16.json`. Kernel facts and the default flip: `tests/tt/results/mk_k4_summary.md`, `mk_k4b_summary.md`,
`mk_k5_summary.md`, `mk_default_summary.md` (+ `mk_tree_20260918-041439.sha256`, the shipped megakernel tree's hashes). Demo golden:
`code/gr00t_p150/demo/n16/expected.json` (fp32 golden + official bf16 reference actions). The RTX 5090 rows are in `GPU_COMPARISON.md`.
The served numbers of this image (`timing_ms` over 50 requests, cold / warm boot) were measured on the build host as described in §2 and
recorded in `docs/publish/build-mk-n16.md` of the port project.