Restore card metadata (license, pipeline_tag, base_model, tags) and upload tt-model authoring files

75e7135 verified 6 days ago

25.7 kB

Serving GR00T N1.5-3B (`nvidia/GR00T-N1.5-3B`) on Blackhole with tt-model-manager

This repo is a tt-model container package source: tt-model.yaml + the code under code/. tt-model package --container builds the OCI image (tt-metal 668c2907575 built from source + the port + the HTTP stack), tt-model serve runs it against the card, tt-model push publishes it to this HF repo. The server is code/gr00t_p150/server/app.py (FastAPI, kind: tt-dit-server); what it runs on the device is the port's Stage-1 path — every op a TTNN op, captured once into four Metal traces (vision, llm, adapter, denoise) and replayed per request; the persistent-megakernel denoise (Stage 2) is not part of this release.


tt-metal tree the image is built from	`/home/deepgadget/experiments/gr00t/publish/tt-metal-668c2907575` — a clean local clone of tt-metal `main` @ `668c290757550588d0ce46b180c344a462a2aaf5` (`v0.79.0-dev20260914-1`), submodules populated, `git status` empty (the working clone `/home/deepgadget/experiments/gr00t/tt-metal` at the same commit has untracked dirs and would be recorded as dirty); torch pin 2.11.0
port code shipped	`code/models/experimental/gr00t/` = tt-metal ref `gr00t-p150-snapshot`, commit `5dc0517ee3bec090b7ead025001a343c2f63aa7b` (parent `668c290`; frozen 2026-09-14T23:36 KST), minus `__pycache__`, `.omc` tool state and 2,001 of the 2,004 per-test result JSONs (the three N1.5 e2e results the card cites and the `d0/d1/d2/mk_k1` summaries stay)
weights	`nvidia/GR00T-N1.5-3B` @ `869830fc749c35f34771aa5209f923ac57e4564e`: `model-0000{1,2,3}-of-00003.safetensors` (5,448,327,040 B), `model.safetensors.index.json`, `config.json`, `experiment_cfg/metadata.json`, `LICENSE` — a pinned pointer, never in the image
app	`gr00t_p150.server.app:app` (uvicorn `--lifespan on`; the lifespan does weights → tokenizer → device → model → trace capture → warm-up → READY)
device recipe	`models.experimental.gr00t.tt.device.open_gr00t_device(trace_region_size=64 MiB, l1_small_size=32768, device_id=0)`; asserts the p150a's 11×10 compute grid and 8 DRAM banks; `TTPolicy(dtype_policy="mixed_dit", trace_layout="per_stage")`
serving shape	embodiment `gr1` (slot 24), layout `gr1`: 1 camera (`ego_view`) × 1 frame × 256 image tokens, ≤ 102 text tokens, LLM sequence padded to 384, state 7/7/6/6 → 16 × [7, 7, 6, 6] actions, batch 1

Below, $PUB=/home/deepgadget/experiments/gr00t/publish (this repo is $PUB/GR00T-N1.5-3B-p150), $TREE=/home/deepgadget/experiments/gr00t/tt-metal (the built working clone, Python 3.10 venv), $ROOT=/home/deepgadget/experiments/tt-models (tt-model-manager tooling).

1. Layout

tt-model.yaml                     authoring manifest (schema 5.1) -- the whole build/serve interface, incl. the model card text
SERVING.md                        this file
GPU_COMPARISON.md                 RTX 5090 vs p150a, matching definitions and caveats
LICENSE                           the NVIDIA License (weights); LICENSE-NOTICE.md says what is under which licence
media/demo_ego_view.png           the demo frame (256x256, GR1 sim robot_sim.PickNPlace traj 0 / step 100)
media/demo_observation.json       the matching raw joint state + instruction (request body minus the image)
media/demo_actions.png            served actions vs the fp32 reference for that request
code/models/experimental/gr00t/   the tt-nn port (tt-metal tree layout): common/ (host pipeline: preprocessing, prompts,
                                  normalisation, checkpoint, weight plan), tt/ (device model, layers, traces, megakernel WIP),
                                  reference/ (fp32 torch reference), tests/, benchmarks/
code/models/{common,tt_dit,demos}/   the 16 tt-metal files the port imports from the tree (source.code in tt-model.yaml)
code/gr00t_p150/                  the policy server: server/{app,schemas,smoke_test}.py, demo/n15/ (canonical demo
                                  observation + golden actions + noise), assets/tokenizer/n15/ (vendored Qwen2 tokenizer)
code/scripts/                     download_weights.sh, bench_http.py
requirements.lock                 the image venv (Python 3.12) as `uv pip freeze` -- written by `tt-model package`
tt_kernel_manifest.json, image/   written by `tt-model package`; uploaded by `tt-model push`

The port keeps its tt-metal-tree absolute imports (models.experimental.gr00t.*, models.tt_dit.*, models.common.*): in the image the code lands at /opt/tt-metal/<path> with PYTHONPATH=/opt/tt-metal, and the image's tt-metal copy excludes models/, so everything under models/ comes from this repo's code/ (the port) and source.code (the 16 tree files). The server package gr00t_p150 is shared with changh95/GR00T-N1.6-3B-p150; this repo ships only the N1.5 assets (demo/n15, assets/tokenizer/n15).

2. Run on the HOST for validation (no Docker)

The tree venv ($TREE/python_env, Python 3.10, torch 2.11.0+cpu, ttnn editable) has everything except fastapi/uvicorn. Do not install into the tree venv; put the HTTP stack in a side directory and prepend it to PYTHONPATH ($PUB/http-side already holds fastapi 0.141.1 / uvicorn 0.53.0 / pydantic 2.13.5 / starlette 1.6.0):

export PATH=$HOME/.local/bin:$PATH
mkdir -p $PUB/http-side && uv pip install --python $TREE/python_env/bin/python --target $PUB/http-side fastapi uvicorn "pydantic>=2"

cd $PUB/GR00T-N1.5-3B-p150
export TT_METAL_HOME=$TREE ARCH_NAME=blackhole
export PYTHONPATH=$PWD/code:$TREE:$TREE/ttnn:$PUB/http-side      # code/ FIRST so models.experimental.gr00t is the snapshot
export TT_METAL_CACHE=$HOME/.cache/tt-metal-cache-gr00t           # warm JIT kernels of the port's device sessions
export HF_MODEL=nvidia/GR00T-N1.5-3B TT_WEIGHTS_REVISION=869830fc749c35f34771aa5209f923ac57e4564e
export GR00T_VERSION=n15 TT_MESH_SHAPE=1x1 TT_DEVICE_ID=0 TT_METAL_VISIBLE_DEVICES=0
export GR00T_TOKENIZER_DIR_N15=$PWD/code/gr00t_p150/assets/tokenizer/n15   # optional: the server picks the vendored dir itself
export GR00T_TT_CACHE=$HOME/.cache/gr00t-tt                                # host .pt plan tier + .tensorbin device tier (~6.4 GB for N1.5)

# import check, no device (what the image's verify.sh does):
$TREE/python_env/bin/python -c "import gr00t_p150.server.app as a, sys; assert a.app and 'ttnn' not in sys.modules; print('ok')"

# serve + smoke + stop -- ONE device-lock hold (the box has one p150a shared with other sessions):
DEVICE_LOCK_TIMEOUT=14400 /home/deepgadget/experiments/gr00t/bin/with-device.sh bash -c '
  '$TREE'/python_env/bin/python -m uvicorn --host 127.0.0.1 --port 20010 --lifespan on gr00t_p150.server.app:app & UV=$!
  python3 code/gr00t_p150/server/smoke_test.py --url http://127.0.0.1:20010 --wait 1800 --out /tmp/gr00t-n15-smoke.json; RC=$?
  kill -TERM $UV; wait $UV; exit $RC'

Boot log landmarks (they drive tt-model serve's checklist): Loading weights: nvidia/GR00T-N1.5-3B @ 869830fc… → Tokenizer files: … → Opening device 0 (trace_region_size=67108864, l1_small_size=32768) → Loading pipeline: Gr00tTT.from_pretrained(n15, …) → Model built in N s (weights N s, 778 tensors, 3176 MB on device, cache path warm|cold) → Warming up: capture per_stage traces on the n15 demo observation, then N traced predict(s) → Warmup k/N: … ms (encode, device, decode) → Warm-up fidelity vs golden fp32: actions PCC … → Warmup complete (capture s, first ms, steady ms; boot s) -- per_stage traces ['vision', 'llm', 'adapter', 'denoise'] → uvicorn Application startup complete. Any failure raises and uvicorn exits non-zero (no CPU fallback); the boot also fails if the warm-up actions' PCC vs the shipped golden is below GR00T_WARMUP_PCC_MIN (0.999). SIGTERM / Ctrl-C: Releasing traces and device tensors → Closing device.

Measured host boots (warm ~/.cache/gr00t-tt and warm TT_METAL_CACHE; $PUB/logs/server_n15_*.log): first boot of a process 14.6 s to READY (model built 9.3 s incl. 9.2 s reading the .tensorbin tier, warm + capture 2.5 s, 5 traced warm-ups 44.4–44.8 ms); an immediately following boot 4.5 s (0.6 s model build with the tiers in the page cache); on the frozen snapshot worktree 5.2 s (server_n15_snapshot_20260914-235713.log). Kernels are rebuilt once for a new TT_METAL_HOME path (~42 s for the 355 kernels of the four traces, docs/publish/snapshot.md §2.3).

Expected smoke line (host, snapshot worktree, $PUB/logs/smoke_n15_snapshot_20260914-235713.log):

PASS GR00T-N1.5-3B-p150 n15: PCC(actions)=0.999982 max|d|=0.0289 PCC(action_pred_valid)=0.999976 repeat_maxdiff=0.0e+00
     seed_path_ok device_ms=42.95/42.16 total_ms=47.22/44.83 wall_ms=50
     per_group[left_arm=0.99995/0.0214 right_arm=0.99997/0.0172 left_hand=0.94335/0.0179 right_hand=0.99999/0.0289]

The per-group PCC / max|d| equal the port's own device test (code/models/experimental/gr00t/tests/tt/results/ test_e2e_predict_actions_vs_golden_n15_20260914-230743.json) to the printed precision; the traced policy is deterministic, so every run reproduces them bit-exactly.

Offline overrides: GR00T_WEIGHTS_DIR=<dir with the snapshot files> (shimmed into a private hub cache so the port's snapshot_dir() resolves to it), GR00T_TOKENIZER_DIR=<dir>.

Offline checks that need no device:

cd $PUB/GR00T-N1.5-3B-p150
# manifest + launcher preview (from the repo dir -- `root: code` is CWD-relative); must print VALID
$ROOT/.venv/bin/python -c "from tt_kernel.container_manifest import load_container_manifest; m = load_container_manifest('tt-model.yaml', check_sources=True); p = m.resolve_profile(); print('VALID', m.name, m.kind, p.hardware, p.mesh_device, m.weights_ref)"
# Python 3.12 resolution of runtime.packages (what the image does) -- must keep numpy<2 and torch 2.11.0+cpu
uv venv --python 3.12 $PUB/depcheck-n15 -q
uv pip install --python $PUB/depcheck-n15/bin/python --dry-run torch==2.11.0 fastapi uvicorn "pydantic>=2" pillow "numpy>=1.24.4,<2" \
   safetensors huggingface_hub "transformers==5.12.1" "opencv-python-headless==4.8.1.78" "torchvision==0.26.0" pytest \
   --extra-index-url https://download.pytorch.org/whl/cpu --index-strategy unsafe-best-match
#   -> torch 2.11.0+cpu, torchvision 0.26.0+cpu, numpy 1.26.4, transformers 5.12.1, opencv-python-headless 4.8.1.78 (cp312 wheel), pytest 9.1.1 (logs/uv-dryrun-n15.log)

3. Package, serve, push (Docker)

Rootless Docker on this box needs source $ROOT/bin/docker-env.sh first (PATH + DOCKER_HOST; the bare docker on PATH is podman). Run every tt-model command from this directory: source.tt_metal and extra_code[].root: code resolve against the process CWD. --out points outside any git checkout because stage() deletes <out>/gr00t-n1.5-3b-p150 before rebuilding. The manifest name is the slug gr00t-n1.5-3b-p150, so that is the build directory, the cache directory (~/.cache/tt-model/gr00t-n1.5-3b-p150/) and the generated card's title.

source $ROOT/bin/docker-env.sh
cd $PUB/GR00T-N1.5-3B-p150

$ROOT/.venv/bin/tt-model package --container tt-model.yaml --out $PUB/build     # tt-metal C++ build (ccache) + venv + verify.sh
# device: serve -> smoke -> bench -> stop inside ONE lock hold (`serve` returns after READY and leaves the container
# running on the card; a second with-device.sh caller would probe a busy card and reset it)
DEVICE_LOCK_TIMEOUT=14400 /home/deepgadget/experiments/gr00t/bin/with-device.sh bash -c '
  source '$ROOT'/bin/docker-env.sh
  '$ROOT'/.venv/bin/tt-model serve '$PUB'/build/gr00t-n1.5-3b-p150/tt_kernel_manifest.json 2>&1 | tee '$PUB'/logs/serve-n15.log || exit 1
  PORT=$(grep -o "127.0.0.1:[0-9]*" '$PUB'/logs/serve-n15.log | head -1 | cut -d: -f2)
  python3 code/gr00t_p150/server/smoke_test.py --url http://127.0.0.1:$PORT --out '$PUB'/logs/smoke-n15.json; RC=$?
  python3 code/scripts/bench_http.py --url http://127.0.0.1:$PORT --reps 50 --warmup 10 --out '$PUB'/logs/bench-http-n15.json
  '$ROOT'/.venv/bin/tt-model logs changh95/GR00T-N1.5-3B-p150 > '$PUB'/logs/container-n15.log 2>&1
  '$ROOT'/.venv/bin/tt-model stop changh95/GR00T-N1.5-3B-p150; exit $RC'
$ROOT/.venv/bin/tt-model push $PUB/build/gr00t-n1.5-3b-p150 --publish

serve pre-downloads the pinned files (a cache hit on this host), then docker run --user 0:0 --device /dev/tenstorrent --ipc host with /dev/hugepages-1G, ~/.cache/huggingface at /hf (rw, HF_HOME=/hf) and ~/.cache/tt-model/gr00t-n1.5-3b-p150/{cache,weights,tensors} at /cache (TT_METAL_CACHE, JIT kernels), /weight-cache (TT_DIT_CACHE_DIR; the port's weight tiers go to /weight-cache/gr00t-tt, ~6.4 GB) and /tensor-cache; it exports exactly HF_MODEL=nvidia/GR00T-N1.5-3B, MESH_DEVICE=P150, TT_MESH_SHAPE=1x1 plus serve.env from tt-model.yaml, publishes the first free port from 20000 and waits ≤ 1800 s for Application startup complete. tt-cli users: tt serve changh95/GR00T-N1.5-3B-p150 / tt model stop changh95/GR00T-N1.5-3B-p150 (after the push).

push makes code/ and image/ on the Hub exactly the staged trees and replaces README.md with the generated card (everything worth keeping lives in card.description / card.quickstart of tt-model.yaml); media/, SERVING.md, GPU_COMPARISON.md, LICENSE, LICENSE-NOTICE.md, tt-model.yaml and .gitattributes at the repo root survive; the license / pipeline_tag / base_model front matter is restored afterwards with huggingface_hub.metadata_update.

4. Container validation (the image built from this manifest)

Image tt-model/gr00t-n1.5-3b-p150:443501bd3b7a (sha256:443501bd3b7afce579bf743eeedd64809886653c29a9d7125614e8451cd61a7b, 3.2 GB as 46 OCI blobs), built 2026-09-14 15:17–15:30 UTC from the manifest above (tt-model package: tt-metal C++ build with a warm ccache ≈ 7 min, Python 3.12 venv, all 17 verify: lines + the launcher's own passed inside the image; 13 min 17 s in total). Validated on the p150a on 2026-09-15 00:56–01:02 KST: three tt-model serve boots of this exact image, each followed by smoke_test.py, a 50-request bench_http.py and tt-model stop, all inside one device-lock hold per run (publish/logs/validate-n15.run2.log = boots 1–2, validate-n15.run3.log = boot 3; per-boot files {serve,smoke,bench-http,container,info}-n15.{2,3}.*; boot 1's per-file artefacts were overwritten by boot 3 and survive only inside validate-n15.run2.log).

boot	caches (`~/.cache/tt-model/gr00t-n1.5-3b-p150/`)	host load	`tt-model serve` → READY	container boot (lifespan)	smoke	`bench_http.py` (10 warm-up + 50): device / server-side median (p90)	`tt-model stop`
1 (cold)	`weights/` and `cache/` empty before; 6.4 GB + 411 MB after	two concurrent tt-metal image builds + CPU tests (load avg 16–110)	1 min 43 s (103.6 s wall)	100.9 s: device open ≈ 4 s → `Model built in 8.9 s (… cache path cold)` (checkpoint → host plan → `.tensorbin`) → trace capture 87.3 s incl. the JIT of the four traces' kernels (`riscv-tt-elf-g++` lines in the container log) → 3 warm-ups 50.9 / 45.8 / 101.5 ms	PASS — PCC(actions) 0.999982, max\|d\| 0.0289; PCC(`action_pred_valid`) 0.999976; device 46.96 / 42.21 ms, total 54.42 / 48.21 ms	44.27 (55.35) / 53.59 (91.12) ms; encode median 3.2, p90 41 ms; client wall 58.1 (94.5)	clean SIGTERM shutdown 4.9 s
2 (warm, immediately after)	reused	same loaded host	20.2 s (21.3 s wall)	15.1 s: `Model built in 0.6 s (… cache path warm)`, capture 6.6 s, warm-ups 47.3 / 119.4 / 65.3 ms (encode spikes = host contention)	PASS — same fidelity; device 50.88 / 42.59 ms, total 67.44 / 45.45 ms	44.51 (55.76) / 58.14 (96.86) ms; encode median 5.8, p90 38 ms; client wall 64.4 (104.2)	clean 3.1 s
3 (warm, quiet host)	reused	load avg ≈ 9, no other device or build work	9.7 s (10.0 s wall)	7.6 s: model 0.6 s, capture 3.2 s, warm-ups 44.5 → 44.3 ms	PASS — same fidelity; device 42.99 / 42.19 ms, total 47.31 / 44.95 ms, client wall 50 ms	42.43 (43.18) / 45.40 (49.20) ms, min 41.66 / 44.46; decode 1.06, encode 1.63, decode_actions 0.14; client wall 47.6 (51.8)	clean 2.0 s

Fidelity is identical in every boot and identical to the host runs of §2 — the served actions JSON of the container is byte-for-byte the host run's (smoke-n15.3.json vs smoke_n15_snapshot_20260914-235713.json) — and equal to the port's device test tests/tt/results/test_e2e_predict_actions_vs_golden_n15_20260914-230743.json to the printed precision: left_arm 0.99995 / 0.0214, right_arm 0.99997 / 0.0172, left_hand 0.94335 / 0.0179 (abs-gated), right_hand 0.99999 / 0.0289. The boot-3 benchmark is the card's "served" row; the device median 42.4 ms is 0.7 ms above the port's own bench_e2e device figure (41.68 ms: upload + traces + readback) and the loaded-host runs show that the extra latency and the p90 tails sit in the host-side stages (encode), not on the device.

GET /info inside the container reports weights.snapshot_dir = /hf/hub/models--nvidia--GR00T-N1.5-3B/snapshots/869830fc… (the mounted HF cache), source.tt_metal = {commit 668c290…, ttnn_dist_version 0.65.2.dev10011}, source.port_snapshot_commit = 5dc0517ee3b…, device_facts grid 11×10 / DRAM 8×1 (info-n15.3.json).

Device-sharing note for this host. The first validation attempt (00:32–00:40 KST) ran while another session's container was still holding the card after that session's lock hold had ended; bin/with-device.sh's probe hung and reset the card twice, then refused to run. Since then every device entry goes through publish/scripts/run_validation_n15_when_free.sh, which waits until docker ps shows no tt-model-* container and the lock is free. Use the manifest path with tt-model stop / logs for an un-pushed package (tt-model stop changh95/GR00T-N1.5-3B-p150 only works after pull).

5. Request / response contract

route	returns
`GET /health`	`{"status": "ok" \| "starting", "model": "GR00T-N1.5-3B-p150", "device": "blackhole:0"}` — 200 always; `ok` only after the warm-up
`GET /info`	model / version / task / hardware / `stage` (`stage1-ttnn-traces` + megakernel note) / `weights` (repo, revision, snapshot dir actually loaded) / `source` (port snapshot commit, server version, tt-metal commit, ttnn dist version) / `inputs` (the full observation contract) / `outputs` / `noise.default_seed` / `limits` / `device_facts` (11×10 grid, 8 DRAM banks) / `warmup_latency_ms` / `warmup_fidelity_vs_golden` / `license`
`GET /v1/models`	OpenAI-shaped stub `{"object": "list", "data": [{"id": "nvidia/GR00T-N1.5-3B", "object": "model", "owned_by": "changh95"}]}` so the tt-model ready card / `tt-model curl` do not 404; this is not a chat API
`GET /demo`	`{"request": <ready-to-post demo body>, "expected": <fp32 reference actions + gates>}`
`POST /predict`	one observation → one 16-step action chunk (below)

POST /predict request (JSON; unknown fields → 422):

field	type	meaning
`images`	`{"ego_view": <b64 PNG/JPEG>}` (or a one-element list per camera)	exactly the GR1 camera; any resolution with sides in [64, 4096] px; the server applies the checkpoint's eval chain (x/255 → 0.95 centre crop → bilinear antialiased 224×224 → Eagle2.5 normalisation), bit-exact vs the reference preprocessing
`state`	`{"left_arm": [[7]], "right_arm": [[7]], "left_hand": [[6]], "right_hand": [[6]]}`	raw physical joint state, one time step (`(D,)` or `(1, D)`), finite floats; sin/cos-encoded and padded to 64 dims on the host
`instruction`	`str` (aliases `language`, `prompt`)	task text, 1–2000 chars, ≤ 102 BPE tokens after the port's prompt build (`repr([instruction])` inside the Eagle2 chat prompt); longer → 400, never truncated
`embodiment`	`"gr1"`, optional	must equal the server's; anything else → 400
`seed`	int ≥ 0, optional	CPU-generator seed of the `[1, 16, 32]` flow-matching noise; default 0 (the deployed policy's)
`noise`	`[[32 floats] × 16]`, optional	explicit initial noise; mutually exclusive with `seed`
`state_dtype`	`"float64"` (default) \| `"float32"`	dtype of the state before normalisation (the GR1 dataset stores float64)
`return_normalized`	bool, default false	also return the model-space chunk `action_pred_normalized` (`[16][32]`) and `action_pred_valid_hd` (`[16, 26]`)

Response (200):

{"actions": {"left_arm": [[7 floats] x 16], "right_arm": [...], "left_hand": [...], "right_hand": [...]},
 "action_horizon": 16, "action_keys": ["left_arm", "right_arm", "left_hand", "right_hand"], "action_dims": {"left_arm": 7, "right_arm": 7, "left_hand": 6, "right_hand": 6},
 "normalized": false, "embodiment": "gr1", "embodiment_id": 24, "version": "n15", "model": "GR00T-N1.5-3B-p150", "layout": "gr1",
 "images": {"ego_view": {"frames": 1, "received_hw": [256, 256]}}, "seq_len": 296, "prompt_tokens": 14, "state_dtype": "float64",
 "noise_source": "seed" | "client", "seed": 0 | null,
 "timing_ms": {"decode": 1.5, "encode": 2.6, "device": 43.0, "decode_actions": 0.1, "total": 47.2}}

actions are physical joint targets (radians) — (y + 1) / 2 · (max − min) + min with the checkpoint's experiment_cfg/metadata.json GR1 statistics — exactly what Gr00tPolicy.get_action returns. timing_ms: decode = base64 + image decode + validation; encode = the port's host preprocessing; device = predict_normalized (static-shape asserts, input writes, 4 × execute_trace, one blocking readback); decode_actions = un-normalisation; total = the whole handler. Errors: 400 structural problems ({"detail": {"errors": [...], "inputs": {...}}}), undecodable image, bad state / instruction / noise, other embodiment; 503 while starting; 500 Type: text on a device failure. Handlers are synchronous and serialised on one lock; batch 1.

6. Environment (read in the lifespan, never at import)

variable	container value (`serve.env`)	meaning
`HF_MODEL` / `TT_WEIGHTS_REVISION`	`nvidia/GR00T-N1.5-3B` (launcher) / `869830fc…`	must be the version's pinned repo and sha (both are asserted against the port's own `configs.HF_REPOS` / `HF_SNAPSHOT_SHAS`)
`GR00T_VERSION`	`n15`	the server code is shared by both GR00T packages and never guesses
`GR00T_EMBODIMENT` / `GR00T_LAYOUT`	`gr1` / `gr1`	the only device-validated layout of this release
`GR00T_POLICY` / `GR00T_TRACE_LAYOUT`	`mixed_dit` / `per_stage`	`TTPolicy` knobs; `bf16` / `mixed` and `two` exist in the port but are not the benchmarked defaults
`GR00T_TRACE_REGION_SIZE` / `GR00T_L1_SMALL_SIZE`	default 67108864 / 32768	the port's validated `open_gr00t_device` values
`GR00T_TOKENIZER_DIR_N15`	`/opt/tt-metal/gr00t_p150/assets/tokenizer/n15`	vendored Qwen2 tokenizer (not in the weights repo); `GR00T_TOKENIZER_DIR` overrides
`GR00T_TT_CACHE`	`/weight-cache/gr00t-tt`	host `.pt` plan tier + `.tensorbin` device tier (persisted under `~/.cache/tt-model/gr00t-n1.5-3b-p150/weights`)
`GR00T_PROJECT_ROOT` / `GR00T_GOLDEN_ROOT`	`/nonexistent`	the port's dev-box defaults; nothing on the serve path needs them
`GR00T_WARMUP_RUNS` / `GR00T_WARMUP_PCC_MIN`	`3` / default 0.999	traced predicts before READY; boot fails below the PCC floor vs the shipped golden
`GR00T_DEFAULT_SEED`, `GR00T_TORCH_THREADS`, `GR00T_WEIGHTS_DIR`	unset	request default seed (0), torch intra-op threads, offline snapshot dir
`TT_MESH_SHAPE`, `TT_DEVICE_ID`, `TT_METAL_VISIBLE_DEVICES`	`1x1` (launcher), `0`, `0`	one chip; any other mesh shape is refused

7. Caveats

Stage-1 path only. TTNN ops in Metal traces; the persistent-megakernel denoise (K1 streaming kernel: 464 GB/s bf16 / 414 GB/s bfp8, code/models/experimental/gr00t/tests/tt/results/mk_k1_summary.md) is not integrated into Gr00tTT and not served.
Shape-locked traces. One embodiment (gr1), one camera, one frame, ≤ 102 text tokens (LLM S padded to 384), batch 1. Other embodiments / cameras / longer instructions → 400 before anything reaches the device. Concurrent requests queue on one lock.
Deterministic noise. Actions are a deterministic function of (observation, seed | noise); the reference noise of the demo is not seed-reproducible, so the smoke test posts it as noise. Same observation + same noise → bit-identical actions.
Base checkpoint. Outputs are the GR1 sim data-config actions of the released N1.5 checkpoint; a real robot needs NVIDIA's post-training.
First container boot is cold: the port converts the 5.4 GB checkpoint into its .pt plan tier and the .tensorbin device tier under /weight-cache/gr00t-tt and JITs the kernels of the four traces into /cache; both persist under ~/.cache/tt-model/gr00t-n1.5-3b-p150/, so later boots are seconds (measured values in §4).
Python 3.12 image vs 3.10 tree venv. Host runs are an approximation; the image is authoritative (its verify.sh imports the app and the port with no device; the served smoke in §4 is the proof).
Weights licence. nvidia/GR00T-N1.5-3B is under the NVIDIA License (non-commercial: research or evaluation use); the server reports it under /info -> license. See LICENSE-NOTICE.md.
tt-model curl and the ready card's /v1/models hint are OpenAI-shaped and are not this API; use the routes above.

Serving GR00T N1.5-3B (nvidia/GR00T-N1.5-3B) on Blackhole with tt-model-manager