# Serving GR00T N1.6-3B (`nvidia/GR00T-N1.6-3B`) on Blackhole with tt-model-manager

This repo is a **tt-model container package source**: `tt-model.yaml` + the code under `code/`. `tt-model package
--container` builds the OCI image (tt-metal from source + the port + the HTTP stack), `tt-model serve` runs it against
the card, `tt-model push` publishes it here. The server is `code/gr00t_p150/server/app.py` (FastAPI, `kind:
tt-dit-server`); the device model is the tt-nn port `code/models/experimental/gr00t` (Stage-1 path: TTNN ops captured
in four Metal traces, DiT matmul weights bfp8_b, everything else bf16). Weights are a pinned pointer, never in the image.

| | |
|---|---|
| tt-metal | main `668c290757550588d0ce46b180c344a462a2aaf5` (`v0.79.0-dev20260914-1-g668c2907575`), built from source inside the image; torch pin 2.11.0 (+cpu in the image) |
| weights | `nvidia/GR00T-N1.6-3B` @ `d0814e7ecb19202e7c8468b46098b0b7ef3a6d61` — `model-0000{1,2}-of-00002.safetensors` (6,573,217,664 B), `model.safetensors.index.json`, `config.json`, `processor_config.json`, `statistics.json`, `embodiment_id.json`, `LICENSE` (NVIDIA License, non-commercial) |
| app | `gr00t_p150.server.app:app` (uvicorn `--lifespan on`; the lifespan resolves the snapshot, opens the device, builds `Gr00tTT`, captures the traces, runs `GR00T_WARMUP_RUNS` traced predictions and checks them against the shipped fp32 golden before READY) |
| device recipe | `models.experimental.gr00t.tt.device.open_gr00t_device(device_id, trace_region_size=64 MiB, l1_small_size=32768, num_command_queues=1)` — the port's validated parameters; asserts the 11×10 compute / 8×1 DRAM grid of a p150 |
| port source | `code/models/experimental/gr00t` = the frozen snapshot commit `5dc0517ee3bec090b7ead025001a343c2f63aa7b` (ref `gr00t-p150-snapshot`, parent = `668c2907575`) of the tt-metal working tree `models/experimental/gr00t`; `code/gr00t_p150` (server) and `code/scripts` were written for this release |
| only served configuration | version `n16`, embodiment `gr1` (slot 20), static layout `gr1` (1 camera × 81 image tokens, `L_max` 24 text tokens, LLM sequence padded to 128), `mixed_dit` dtype policy, `per_stage` trace layout, batch 1 |

Below, `$ROOT` = the tt-models tooling checkout (`/home/deepgadget/experiments/tt-models` on the build host: `tt-model` CLI in
`$ROOT/.venv/bin`, rootless Docker env in `$ROOT/bin/docker-env.sh`), `$TREE` = the tt-metal tree with the built `python_env`
(`/home/deepgadget/experiments/gr00t/tt-metal`), `$REPO` = this directory.

## Layout

```
tt-model.yaml                 authoring manifest (schema 5.1) -- the whole build/serve interface
tt_kernel_manifest.json       GENERATED wire manifest: weights pointer, image tag/digest, runtime, serve, verify, built provenance
requirements.lock             GENERATED `pip freeze` of the image venv (Python 3.12, torch 2.11.0+cpu, ttnn <scm>)
image/                        GENERATED OCI layout of the image (~3 GB)
README.md                     the card (generated by `tt-model package` from card.description / card.quickstart; front matter restored after push)
SERVING.md, GPU_COMPARISON.md, LICENSE, LICENSE-NOTICE.md     authored
media/demo_ego_view.png       the GR1 PickNPlace demo frame (256x256), media/demo_observation.json the matching raw state + instruction
media/demo_actions_vs_golden.png   served action chunk vs the fp32 reference, one panel per action dimension
code/models/experimental/gr00t/    the port: common/ (host preprocessing, prompts, normalisation, checkpoint, weight plan),
                                   reference/ (fp32 torch reference), tt/ (device model, layers, traces, tt/megakernel/ Stage-2 work),
                                   tests/ (CPU + device suites and their results JSONs), benchmarks/ (bench_e2e etc. + results)
code/models/common/utility_functions.py, code/models/tt_dit/{utils,layers,parallel,encoders/qwen3vl}/..., code/models/demos/deepseek_v3_b1/unified_kernel_descriptor.py
                                   the tt-metal files the port imports from the tree other than itself (staged by tt-model from `source.code`)
code/gr00t_p150/              server/{app.py, schemas.py, smoke_test.py}, demo/n16/ (canonical demo observation + golden), assets/tokenizer/n16/ (vendored Qwen2 tokenizer)
code/scripts/                 download_weights.sh (pinned snapshot -> HF cache), bench_http.py (HTTP latency, 50 requests)
```

In the image the code lands at `/opt/tt-metal/<path>` with `PYTHONPATH=/opt/tt-metal`, which is exactly the tree layout the port was
written for (`models.experimental.gr00t.*`, `models.tt_dit.*` absolute imports; `models` is a namespace package).

## 1. Run on the HOST (hardware validation, no Docker)

The tree venv `$TREE/python_env` (Python 3.10, torch 2.11.0+cpu, ttnn editable, transformers 5.12.1, opencv 4.8.1) has everything
except `fastapi` / `uvicorn`. Do not install into the tree venv; put the HTTP stack in a side directory and prepend it to `PYTHONPATH`.
`$REPO/code` must be FIRST on `PYTHONPATH` so `models.experimental.gr00t` resolves to the shipped snapshot (the tree's `models/` is still
needed for `models.tt_dit.*` on the host; in the image the staged `code/` is the only `models` tree).

```bash
export TREE=/home/deepgadget/experiments/gr00t/tt-metal
export PATH=$HOME/.local/bin:$PATH                                       # uv
HTTP=/home/deepgadget/experiments/gr00t/publish/http-side                # any writable dir outside the trees
uv pip install --python $TREE/python_env/bin/python --target $HTTP fastapi uvicorn "pydantic>=2"

cd $REPO
export PYTHONPATH=$PWD/code:$TREE:$TREE/ttnn:$TREE/tools:$HTTP
export TT_METAL_HOME=$TREE ARCH_NAME=blackhole
export TT_METAL_CACHE=$HOME/.cache/tt-metal-cache-gr00t                  # JIT kernels (7 GB warm on the build host)
export HF_MODEL=nvidia/GR00T-N1.6-3B
export TT_WEIGHTS_REVISION=d0814e7ecb19202e7c8468b46098b0b7ef3a6d61      # scripts/download_weights.sh puts it in ~/.cache/huggingface
export TT_MESH_SHAPE=1x1 TT_DEVICE_ID=0 TT_METAL_VISIBLE_DEVICES=0
export GR00T_VERSION=n16 GR00T_EMBODIMENT=gr1 GR00T_LAYOUT=gr1 GR00T_POLICY=mixed_dit GR00T_TRACE_LAYOUT=per_stage
export GR00T_TOKENIZER_DIR=$PWD/code/gr00t_p150/assets/tokenizer/n16
export GR00T_TT_CACHE=$HOME/.cache/gr00t-tt                              # host .pt plan tier + .tensorbin device tier (~8 GB for n16)
export GR00T_WARMUP_RUNS=3

# import check, no device (what the image's verify.sh does):
$TREE/python_env/bin/python -c "import gr00t_p150.server.app as a; assert a.app; print('ok')"

# serve (opens the device, loads the weights, captures the four traces, 3 warm-up predictions checked against the golden, then READY).
# The one Blackhole card of the build host is shared with other agents: run server + client + shutdown inside ONE lock hold.
DEVICE_LOCK_TIMEOUT=14400 /home/deepgadget/experiments/gr00t/bin/with-device.sh bash -c '
  $TREE/python_env/bin/python -m uvicorn --host 127.0.0.1 --port 20016 --lifespan on gr00t_p150.server.app:app & UV=$!
  python3 code/gr00t_p150/server/smoke_test.py --url http://127.0.0.1:20016 --wait 1800 --out /tmp/gr00t-n16-smoke.json; RC=$?
  python3 code/scripts/bench_http.py --url http://127.0.0.1:20016 --n 50 --out /tmp/gr00t-n16-bench.json
  kill -TERM $UV; wait $UV; exit $RC'
```

Boot log landmarks (they drive `tt-model serve`'s boot checklist): `Loading weights: nvidia/GR00T-N1.6-3B @ d0814e7e…` →
`Tokenizer files: …` → `Opening device 0 (trace_region_size=67108864, l1_small_size=32768)` → `Loading pipeline: Gr00tTT.from_pretrained(...)`
→ `Model built in N s (1037 tensors, 3775 MB on device, cache path warm|cold)` → `Warming up: capture per_stage traces on the n16 demo
observation, then 3 traced predict(s)` → `Warmup k/3: … ms` → `Warm-up fidelity vs golden fp32: actions PCC 0.9992…` → `Warmup complete (…)`
→ uvicorn `Application startup complete`. Startup failures raise and uvicorn exits non-zero (no CPU fallback); a warm-up PCC below
`GR00T_WARMUP_PCC_MIN` (0.999) fails the boot. Stop with SIGTERM / Ctrl-C: the lifespan releases the traces and device tensors and closes
the device (`Releasing traces and device tensors`, `Closing device`).

Expected smoke line on the demo observation (host, warm caches, `publish/logs/smoke_n16_snapshot_20260914-235724.log` of the build host):

```
PASS GR00T-N1.6-3B-p150 n16: PCC(actions)=0.999209 max|d|=0.0971 PCC(action_pred_valid)=0.999213 repeat_maxdiff=0.0e+00 seed_path_ok device_ms=57.14/57.03 total_ms=64.11/62.68 wall_ms=70 per_group[left_arm=0.99995/0.0176 right_arm=0.99991/0.0423 left_hand=0.80114/0.0589 right_hand=0.99312/0.0971 waist=0.77216/0.0170]
```

Host boot with warm `GR00T_TT_CACHE` + `TT_METAL_CACHE`: model built 0.6–8.5 s (page cache), capture 1.7–2.0 s, 5 warm-ups ~60 ms each,
READY after 5–13 s. Offline overrides: `GR00T_WEIGHTS_DIR=<dir with the seven snapshot files>` (shimmed into a private hub cache so the
port's `configs.snapshot_dir("n16")` resolves to it); `GR00T_TOKENIZER_DIR=<dir>`. Every variable is read in the lifespan, never at import.

## 2. Package, serve, push (Docker)

Rootless Docker on the build host needs `source $ROOT/bin/docker-env.sh` first (PATH + `DOCKER_HOST`; the bare `docker` is podman).
**Run every `tt-model` command from this directory**: `source.tt_metal` and `extra_code[].root: code` resolve against the process CWD.
`--out` points outside any git checkout because `stage()` deletes `<out>/gr00t-n1.6-3b-p150` before rebuilding.

```bash
source $ROOT/bin/docker-env.sh
cd $REPO
# offline validation (must print VALID)
$ROOT/.venv/bin/python -c "from tt_kernel.container_manifest import load_container_manifest as L; m = L('tt-model.yaml', check_sources=True); print('VALID', m.name, m.weights_ref)"

$ROOT/.venv/bin/tt-model package --container tt-model.yaml --out /home/deepgadget/experiments/gr00t/publish/build   # ~2 h cold (tt-metal C++ build), runs verify.sh
# serve -> smoke -> bench -> stop inside ONE device-lock hold (the container keeps the card after `serve` returns)
DEVICE_LOCK_TIMEOUT=14400 /home/deepgadget/experiments/gr00t/bin/with-device.sh bash -c '
  source $ROOT/bin/docker-env.sh
  $ROOT/.venv/bin/tt-model serve /home/deepgadget/experiments/gr00t/publish/build/gr00t-n1.6-3b-p150/tt_kernel_manifest.json || exit 1
  python3 code/gr00t_p150/server/smoke_test.py --url http://127.0.0.1:20000 --out /tmp/gr00t-n16-smoke.json
  python3 code/scripts/bench_http.py --url http://127.0.0.1:20000 --n 50 --out /tmp/gr00t-n16-bench.json
  $ROOT/.venv/bin/tt-model stop /home/deepgadget/experiments/gr00t/publish/build/gr00t-n1.6-3b-p150/tt_kernel_manifest.json'   # org/name is a valid target only after pull/push
$ROOT/.venv/bin/tt-model push /home/deepgadget/experiments/gr00t/publish/build/gr00t-n1.6-3b-p150 --publish
```

`serve` pre-downloads the pinned files into `~/.cache/huggingface` (a metadata no-op when present), then
`docker run --user 0:0 --device /dev/tenstorrent --ipc host --mount /dev/hugepages-1G -v ~/.cache/huggingface:/hf
-v ~/.cache/tt-model/gr00t-n1.6-3b-p150/{cache,weights,tensors}:/{cache,weight-cache,tensor-cache} -p 20000:20000` with exactly
`HF_MODEL=nvidia/GR00T-N1.6-3B`, `MESH_DEVICE=P150`, `TT_MESH_SHAPE=1x1`, `HF_HOME=/hf`, `TT_METAL_CACHE=/cache`,
`TT_DIT_CACHE_DIR=/weight-cache`, `TT_CACHE_PATH=/tensor-cache` and the `serve.env` block of `tt-model.yaml`
(`TT_WEIGHTS_REVISION`, `GR00T_VERSION=n16`, `GR00T_TT_CACHE=/weight-cache/gr00t-tt`, `GR00T_TOKENIZER_DIR=/opt/tt-metal/gr00t_p150/assets/tokenizer/n16`, …),
and waits ≤ 1800 s for `Application startup complete`. Serves on 20000 or the next free port (printed). tt-cli users after the push:
`tt serve changh95/GR00T-N1.6-3B-p150` / `tt model stop changh95/GR00T-N1.6-3B-p150`.

Caches on the host (persist across boots): `~/.cache/tt-model/gr00t-n1.6-3b-p150/cache` (JIT kernels, `TT_METAL_CACHE`),
`~/.cache/tt-model/gr00t-n1.6-3b-p150/weights/gr00t-tt/n16/...` (the port's host plan tier + `.tensorbin` device tier, ~8 GB, written on
the first boot), weights in `~/.cache/huggingface/hub/models--nvidia--GR00T-N1.6-3B`.

Measured on the build host with this image (`docs/publish/build-n16.md` has every command and log):

| boot | wall from `docker run` to READY | notes |
|---|---:|---|
| first (cold: empty `/cache` and `/weight-cache`) | 137 s | checkpoint → device-layout plan + `.tensorbin` write, JIT of every kernel of the four traces, trace capture, 3 warm-ups (round-0 build `7e1e1380a5a5`, 2026-09-15 00:30 KST) |
| second (warm caches, 15 min later) | 13 s | `Model built in 1.2 s` (weight tier in the host page cache), capture 5.2 s (round-0 build) |
| third (warm caches, 1 h later, the shipped build `feb77048a2ed`) | 21 s | `Model built in 11.2 s` (the 7.9 GB weight tier re-read from disk), capture 4.6 s, 3 warm-ups 60.8 / 59.4 / 59.9 ms; smoke PASS, 50-request bench 56.8 / 59.4 ms, clean stop 2.1 s |

The shipped image (`feb77048a2ed`, fix round 1) differs from the round-0 build only in the server's `GET /info → license` text and in
`code/` shipping 8 instead of 2,009 `tests/tt/results` files; the tt-metal build, the port, the serve path and the JIT / weight caches are
identical, so the cold-boot figure was not re-measured. The served latency in the card is the round-1 run of the shipped image.

`push` makes `code/` and `image/` on the Hub exactly the staged trees and replaces `README.md` with the generated card (everything worth
keeping lives in `card.description` / `card.quickstart`); `tt-model.yaml`, `SERVING.md`, `GPU_COMPARISON.md`, `LICENSE`,
`LICENSE-NOTICE.md`, `media/` survive (re-uploaded after every push). The front matter (`license: other`, `license_name: nvidia-license`,
`license_link`, `pipeline_tag: robotics`, `base_model`, extra tags) is restored afterwards with `huggingface_hub.metadata_update`.

## 3. Request / response contract

| route | returns |
|---|---|
| `GET /health` | `{"status": "ok" \| "starting", "model": "GR00T-N1.6-3B-p150", "device": "blackhole:0"}` — 200 always; `ok` only after warm-up |
| `GET /info` | model / version / task / hardware / `stage` (`served: stage1-ttnn-traces` + megakernel note) / `weights` (repo, revision, snapshot dir actually loaded) / `source` (port snapshot commit, server version, tt-metal commit, ttnn version) / `inputs` (the full observation contract) / `outputs` / `noise.default_seed` (42) / `limits` / `device_facts` / `warmup_latency_ms` / `warmup_fidelity_vs_golden` / `license` |
| `GET /demo` | `{"request": <ready-to-post demo body>, "expected": <golden actions + gates>}` |
| `GET /v1/models` | OpenAI-shaped stub (`{"object": "list", "data": [{"id": "nvidia/GR00T-N1.6-3B", ...}]}`) so the tt-model ready card does not 404; this is not a chat API |
| `POST /predict` | one observation → one 16-step action chunk (below) |

`POST /predict` request (JSON; unknown fields → 422):

| field | type | meaning |
|---|---|---|
| `images` | `{camera key: base64 PNG/JPEG}` or `{camera key: [one frame]}` | exactly one camera, `ego_view_bg_crop_pad_res256_freq20`; any resolution with sides in [64, 4096]; the frame goes through the version's own eval chain (letterbox → 256 INTER_AREA → 0.95 centre crop → 256 → PIL bicubic 252 → Eagle3 normalisation), so no client-controlled shape reaches ttnn |
| `state` | `{group: [D floats]}` (or `[[D floats]]`) | raw physical joint state: `left_arm` 7, `right_arm` 7, `left_hand` 6, `right_hand` 6, `waist` 3; finite |
| `instruction` | str (aliases `language`, `prompt`) | task text, 1–2000 chars; ≤ 24 BPE tokens in this layout (longer → 400: `n_text exceeds L_max`); leading/trailing whitespace stripped |
| `embodiment` | str, optional | must be `gr1` |
| `seed` | int ≥ 0, optional | CPU-generator seed of the `[1, 50, 128]` flow-matching noise; default 42 (the deployed policy's seed) |
| `noise` | `[50, 128]` or `[1, 50, 128]` floats, optional | explicit initial noise (the smoke test sends the reference noise); exclusive with `seed` |
| `state_dtype` | `float32` \| `float64`, optional | default `float32` for N1.6 (the reference dataset's dtype) |
| `return_normalized` | bool, optional | also return `action_pred_normalized` (`[50, 128]` model-space chunk) and `action_pred_valid_hd` (`[16, 29]`) |

Response (200):

```json
{"actions": {"left_arm": [[7 floats] x 16], "right_arm": [...], "left_hand": [[6] x 16], "right_hand": [...], "waist": [[3] x 16]},
 "action_horizon": 16, "action_keys": ["left_arm", "right_arm", "left_hand", "right_hand", "waist"], "action_dims": {"left_arm": 7, "right_arm": 7, "left_hand": 6, "right_hand": 6, "waist": 3},
 "normalized": false, "embodiment": "gr1", "embodiment_id": 20, "version": "n16", "model": "GR00T-N1.6-3B-p150", "layout": "gr1",
 "images": {"ego_view_bg_crop_pad_res256_freq20": {"frames": 1, "received_hw": [256, 256]}}, "seq_len": 116, "prompt_tokens": 12,
 "state_dtype": "float32", "noise_source": "seed", "seed": 42,
 "timing_ms": {"decode": 4.0, "encode": 2.8, "device": 57.1, "decode_actions": 0.2, "total": 64.1}}
```

`actions` are physical, un-normalised joint targets (float32 as decimal JSON): the arm and hand groups are the checkpoint's RELATIVE
outputs already composed onto the request's `state` (per-step min/max un-normalisation from `statistics.json` + the raw last state,
clipped), `waist` is ABSOLUTE — exactly `Gr00tPolicy.get_action` for the GR1 data config. `timing_ms`: `decode` = base64 + PNG decode +
validation, `encode` = the port's host preprocessing, `device` = `predict_normalized` (input writes, `execute_trace` × 4, one blocking
readback), `decode_actions` = un-normalisation, `total` = handler wall time. Errors: **400** (wrong/missing camera or state group, wrong
state width, non-finite state, undecodable image, frame outside [64, 4096], empty/too-long instruction, other embodiment, `seed` +
`noise` together, malformed noise), **422** (schema), **503** while starting, **500** `Type: text` on a device failure. Handlers are
synchronous and serialised on one lock; batch is 1.

## 4. Caveats

* **Stage-1 path.** TTNN ops in four Metal traces (`vision` 11.5 ms, `llm` 9.5, `adapter` 2.0, `denoise` 30.4 — `bench_e2e` medians,
  `code/models/experimental/gr00t/benchmarks/results/e2e_stage1_n16.json`); DiT matmul weights bfp8_b, everything else bf16. The
  persistent-megakernel denoise (Stage 2, `tt/megakernel/`, rung K1: 464 GB/s bf16 / 414 GB/s bfp8 weight streaming,
  `tests/tt/results/mk_k1_summary.md`) is **not** integrated into `Gr00tTT` and not served.
* **N1.6 fidelity margin.** Final actions reproduce the fp32 golden at PCC 0.99921 on the normalised valid slice (the official bf16 GPU
  path scores 0.99956 there; boot/smoke gate 0.999). Four intermediate DiT taps miss their per-tap gates on the state-token row 0 only
  (`dit_out[k=1]` 0.999556 vs 0.99961, `dit_out[k=2]` 0.999496 vs 0.99964, `action_decoder_out[k=1]` 0.999579 vs 0.9997,
  `action_decoder_out[k=2]` 0.999556 vs 0.99978; 52/56 gated rows pass (57/61 rows incl. the determinism row and the four xfail info rows) —
  `tests/tt/results/test_e2e_untraced_taps_vs_golden_n16_20260914-232434.json`, `tests/tt/results/d2_summary.md` §3.2). Nothing downstream
  reads row 0; every action tap passes.
* **One embodiment, one layout, one camera, batch 1.** `gr1` (GR1 arms + hands + waist), static layout `gr1` (81 image tokens, `L_max` 24,
  `S_pad` 128). `GR00T_LAYOUT=gr1_long_text` (`L_max` 152, `S_pad` 256) exists in the port but was **not** device-validated. Other
  embodiments need their own layout + validation.
* **Deterministic noise.** Actions are a deterministic function of (observation, seed | noise); the default seed 42 reproduces the
  reference `initial_noise` bit-exactly (`demo/n16/noise.json: seed_equivalent`). Two identical requests return bit-identical chunks.
* **First boot is cold**: the checkpoint is converted to the device layout into `/weight-cache` (~8 GB) and every kernel is JIT-compiled
  into `/cache`; both persist under `~/.cache/tt-model/gr00t-n1.6-3b-p150/`. Measured boot times: §2.
* **Python 3.12 image vs 3.10 host venv.** The image resolves the packages on 3.12 (`requirements.lock`); the host venv is 3.10. The
  code is 3.10/3.12 compatible; the image is the authoritative validation (§2 numbers).
* **Vendored tokenizer** (`code/gr00t_p150/assets/tokenizer/n16`, NVIDIA License — see `LICENSE-NOTICE.md`): verified bit-exact against the
  Isaac-GR00T checkout and the golden `input_ids`.
* `tt-model curl` / `GET /v1/models` are OpenAI-shaped and are not this API; use the routes above.

## 5. Where the numbers come from

Everything in the card is measured and shipped: `code/models/experimental/gr00t/benchmarks/results/e2e_stage1_n16.json` (Stage-1 latency,
50 calls, tt-metal `668c2907575`), `code/models/experimental/gr00t/tests/tt/results/test_e2e_{predict_actions_vs_golden,
traced_equals_untraced,untraced_taps_vs_golden}_n16_20260914-2324*.json` (fidelity, determinism, the four marginal taps),
`tests/tt/results/d2_summary.md` (device session D2), `tests/tt/results/mk_k1_summary.md` (megakernel K1), `code/gr00t_p150/demo/n16/expected.json`
(fp32 golden + official bf16 reference actions for the demo step). The RTX 5090 rows are in `GPU_COMPARISON.md`. The served numbers of this
image (`timing_ms` over 50 requests, cold / warm boot) were measured on the build host as described in §2 and recorded in
`docs/publish/build-n16.md` of the port project.