tt-model.yaml

# SPDX-License-Identifier: Apache-2.0
# tt-model-manager container manifest (schema 5.1) for NVIDIA Isaac GR00T N1.5-3B (Stage-1 TTNN path) on Blackhole.
#
# Run EVERY tt-model command from this directory: `source.tt_metal` and `extra_code[].root: code`
# resolve against the process CWD, not against this file.  Rootless Docker on this box needs
# `source /home/deepgadget/experiments/tt-models/bin/docker-env.sh` first (bare `docker` is podman).
#
#   tt-model package --container tt-model.yaml --out /home/deepgadget/experiments/gr00t/publish/build
#   DEVICE_LOCK_TIMEOUT=14400 /home/deepgadget/experiments/gr00t/bin/with-device.sh bash -c '
#       tt-model serve /home/deepgadget/experiments/gr00t/publish/build/gr00t-n1.5-3b-p150/tt_kernel_manifest.json && \
#       python3 code/gr00t_p150/server/smoke_test.py --url http://127.0.0.1:20000; tt-model stop changh95/GR00T-N1.5-3B-p150'
#   tt-model push /home/deepgadget/experiments/gr00t/publish/build/gr00t-n1.5-3b-p150 --publish
schema: "5.1"

repo: changh95/GR00T-N1.5-3B-p150
name: gr00t-n1.5-3b-p150          # slug ([a-z0-9][a-z0-9._-]*): cache dir, container name, generated H1

# A POINTER, pinned.  Weights are never baked into the image: `tt-model serve` pre-downloads exactly these files
# at this sha into the host HF cache (mounted at /hf, HF_HOME=/hf) and the server resolves the same snapshot with
# huggingface_hub.snapshot_download(revision=TT_WEIGHTS_REVISION, local_files_only=True) and the port's own
# common.configs.snapshot_dir() (both must agree, the server asserts it).
weights:
  repo: nvidia/GR00T-N1.5-3B
  revision: 869830fc749c35f34771aa5209f923ac57e4564e
  allow_patterns:
    - "model-*.safetensors"            # 3 bf16 shards, 5,448,327,040 bytes
    - model.safetensors.index.json     # required by the port's LazyCheckpoint (common/checkpoint.py)
    - config.json
    - experiment_cfg/metadata.json     # GR1 normalisation statistics (common/normalization.py)
    - LICENSE                          # NVIDIA License -- keep it next to the weights it covers

kind: tt-dit-server
arch: blackhole

source:
  # Clean checkout of tt-metal main 668c2907575 (v0.79.0-dev20260914-1): a local clone of the validated tree with its
  # three submodules populated and `git status --porcelain` empty, so the image is built from exactly the commit the
  # card links to (the working clone has untracked dirs and would be recorded as a dirty tree).
  tt_metal: /home/deepgadget/experiments/gr00t/publish/tt-metal-668c2907575
  # tt-metal-relative files the port imports from the tree OTHER than itself, with their import closure
  # (the image's tt-metal copy excludes models/, so everything under models/ must be listed here or in extra_code).
  code:
    - models/common/utility_functions.py                         # models.tt_dit.utils.matmul imports is_blackhole (needs pytest!)
    - models/tt_dit/utils/matmul.py                              # tt/layers.py: the `minimal` matmul rows (lazy)
    - models/tt_dit/utils/agmm_rules.py                          # lazy from matmul.py
    - models/tt_dit/utils/mmrs_rules.py                          # lazy from matmul.py
    - models/tt_dit/utils/tensor.py                              # layers/linear.py, layers/module.py, lora.py, parallel/manager.py
    - models/tt_dit/utils/progress.py                            # layers/module.py
    - models/tt_dit/utils/walltime.py                            # utils/progress.py
    - models/tt_dit/utils/substate.py                            # layers/module.py
    - models/tt_dit/layers/linear.py                             # encoders/qwen3vl/vision_qwen3vl.py (N1.7 tower; lazy, unused by N1.5)
    - models/tt_dit/layers/module.py
    - models/tt_dit/layers/normalization.py
    - models/tt_dit/layers/lora.py                               # imported at the bottom of layers/linear.py
    - models/tt_dit/parallel/config.py
    - models/tt_dit/parallel/manager.py
    - models/tt_dit/encoders/qwen3vl/vision_qwen3vl.py           # tt/qwen3vl_vision.py (lazy; not on the N1.5 serve path)
    - models/demos/deepseek_v3_b1/unified_kernel_descriptor.py   # tt/megakernel/descriptors.py (lazy; Stage-2 work, not served)
  # This repo's own code (lands at /opt/tt-metal/<path>; PYTHONPATH=/opt/tt-metal).
  extra_code:
    - root: code
      paths:
        - models          # code/models/experimental/gr00t: the frozen snapshot of the port (tt-metal ref gr00t-p150-snapshot, 5dc0517ee3b)
        - gr00t_p150      # the policy server, its smoke test, the demo observation and the vendored Qwen2 tokenizer files
        - scripts         # download_weights.sh, bench_http.py
  ubuntu: "22.04"
  python: "3.12"

runtime:
  app: gr00t_p150.server.app:app
  mesh_shape_env: TT_MESH_SHAPE
  # On top of the kind defaults (fastapi, uvicorn, pydantic>=2, pillow), the auto-pinned torch==2.11.0+cpu (the tree's
  # requirements-dev.txt pin) and ttnn's own deps (numpy<2, loguru, ...).  Every third-party module on the serve path,
  # pinned where the tree pins it.  Resolution on Python 3.12 proven with a uv dry-run (SERVING.md).
  packages:
    - "numpy>=1.24.4,<2"
    - safetensors                        # common/checkpoint.py
    - huggingface_hub                    # server: snapshot_download(local_files_only=True) guard
    - transformers==5.12.1               # common/prompts.get_tokenizer (Qwen2 tokenizer, lazy); tree pin
    - opencv-python-headless==4.8.1.78   # common/preprocessing.py imports cv2 at module level; tree pin (cp312 wheel exists)
    - torchvision==0.26.0                # preprocessing.n15_crop_resize (center_crop / resize antialias); tree pin for torch 2.11.0
    - pytest                             # models/common/utility_functions.py imports pytest at module level

serve:
  port: 20000
  hardware: p150
  mesh_device: P150
  env:
    TT_WEIGHTS_REVISION: "869830fc749c35f34771aa5209f923ac57e4564e"   # == weights.revision == configs.HF_SNAPSHOT_SHAS["n15"]
    TT_METAL_VISIBLE_DEVICES: "0"
    TT_DEVICE_ID: "0"
    GR00T_VERSION: "n15"
    GR00T_EMBODIMENT: "gr1"              # the only embodiment with a device-validated layout in this release
    GR00T_LAYOUT: "gr1"                  # static layout: 1 image x 256 tokens, L_max 102 text tokens, LLM sequence padded to 384
    GR00T_POLICY: "mixed_dit"            # TTPolicy.dtype_policy: DiT block matmul weights bfp8_b, everything else bf16 (validated default)
    GR00T_TRACE_LAYOUT: "per_stage"      # four Metal traces vision / llm / adapter / denoise (the benchmarked layout)
    GR00T_TOKENIZER_DIR_N15: "/opt/tt-metal/gr00t_p150/assets/tokenizer/n15"
    GR00T_PROJECT_ROOT: "/nonexistent"   # the port's dev-box default for project-relative paths; nothing on the serve path needs it
    GR00T_GOLDEN_ROOT: "/nonexistent"    # tests only (defensive)
    GR00T_TT_CACHE: "/weight-cache/gr00t-tt"   # host plan (.pt) + device-layout (.tensorbin) weight tiers -> ~/.cache/tt-model/gr00t-n1.5-3b-p150/weights
    GR00T_WARMUP_RUNS: "3"               # traced predict() calls after trace capture, before READY

# Build-time assertions, run INSIDE the finished image as uid 1000, no device, no weights, no HF_MODEL.
verify:
  - "import gr00t_p150.server.app as a; assert a.app; import sys; assert 'ttnn' not in sys.modules, 'server import must be side-effect free'"
  - "import models.experimental.gr00t.tt.model as m; assert m.Gr00tTT"
  - "from models.experimental.gr00t.common import preprocessing, normalization, prompts, checkpoint, weights, configs; assert configs.HF_SNAPSHOT_SHAS['n15'] == '869830fc749c35f34771aa5209f923ac57e4564e'"
  - "import pytest, models.common.utility_functions; import models.tt_dit.utils.matmul as mm; assert mm.get_matmul_config"
  - "import models.demos.deepseek_v3_b1.unified_kernel_descriptor as u; assert u.UnifiedKernelDescriptor"
  - "import models.tt_dit.encoders.qwen3vl.vision_qwen3vl"
  - "import models.experimental.gr00t.tt.layers, models.experimental.gr00t.tt.qwen3vl_vision, models.experimental.gr00t.tt.megakernel.descriptors"
  - "import safetensors, huggingface_hub, cv2, numpy, PIL; assert int(numpy.__version__.split('.')[0]) < 2, numpy.__version__"
  - "import transformers; assert transformers.__version__ == '5.12.1', transformers.__version__"
  - "import torchvision; assert torchvision.__version__.startswith('0.26.0'), torchvision.__version__"
  - "from pathlib import Path; d = Path('/opt/tt-metal/gr00t_p150/assets/tokenizer/n15'); assert all((d / f).is_file() for f in ('vocab.json', 'merges.txt', 'tokenizer_config.json', 'special_tokens_map.json', 'added_tokens.json')), sorted(p.name for p in d.iterdir())"
  - "import os; os.environ['GR00T_TOKENIZER_DIR_N15'] = '/opt/tt-metal/gr00t_p150/assets/tokenizer/n15'; from models.experimental.gr00t.common.prompts import get_tokenizer; t = get_tokenizer('n15'); assert t.convert_tokens_to_ids('<IMG_CONTEXT>') == 151669"
  - "from pathlib import Path; assert Path('/opt/tt-metal/models/experimental/gr00t/common/golden_tap_map.json').is_file()"
  - "from pathlib import Path; assert Path('/opt/tt-metal/gr00t_p150/server/smoke_test.py').is_file() and Path('/opt/tt-metal/scripts/bench_http.py').is_file()"
  - "from pathlib import Path; d = Path('/opt/tt-metal/gr00t_p150/demo/n15'); assert all((d / f).is_file() for f in ('observation.json', 'noise.json', 'expected.json', 'frames/ego_view_t0.png')), sorted(str(p) for p in d.rglob('*'))"
  - "import ttnn; assert all(hasattr(ttnn, a) for a in ('begin_trace_capture', 'end_trace_capture', 'execute_trace', 'release_trace', 'copy_host_to_device_tensor', 'MinimalMatmulConfig'))"
  - "from gr00t_p150.server.app import parse_mesh_shape as p; assert p('1x1') == p('(1, 1)') == p('1,1') == (1, 1)"

card:
  description: >
    NVIDIA Isaac GR00T N1.5 (3B vision-language-action policy: SigLIP ViT (224², 256 tokens/image) → linear projector →
    12-layer Qwen3-1.7B backbone → 4-block VL self-attention → 16-block cross/self AdaLN DiT, 4 Euler flow-matching
    steps) running on one Tenstorrent Blackhole p150a via tt-nn: one camera frame + proprioceptive state + a task
    instruction in, a 16-step chunk of GR1 arm/hand joint targets out. Stage-1 release: every op is a TTNN op replayed
    from four Metal traces (DiT block matmul weights bfp8, everything else bf16); the persistent-megakernel denoise is in
    progress and not in this image.

    Weights: [nvidia/GR00T-N1.5-3B](https://huggingface.co/nvidia/GR00T-N1.5-3B) (NVIDIA License, non-commercial) ·
    Paper: [arXiv:2503.14734](https://arxiv.org/abs/2503.14734) (GR00T N1 report; the N1.5 changes are described in the
    [N1.5 model card](https://huggingface.co/nvidia/GR00T-N1.5-3B)) ·
    Upstream code: [NVIDIA/Isaac-GR00T](https://github.com/NVIDIA/Isaac-GR00T) (tag `n1.5-release`) ·
    Port: `code/models/experimental/gr00t` in this repo (tt-metal-tree layout, frozen snapshot `5dc0517ee3b`) + the
    policy server `code/gr00t_p150`
  quickstart: |
    ### Run with tt-cli

    ```bash
    tt serve changh95/GR00T-N1.5-3B-p150
    python3 - <<'EOF'
    import base64, json
    body = json.load(open("media/demo_observation.json"))          # GR1 PickNPlace demo step: raw joint state + instruction
    body.pop("_note", None)
    body["images"] = {"ego_view": base64.b64encode(open("media/demo_ego_view.png", "rb").read()).decode()}
    json.dump(body, open("req.json", "w"))
    EOF
    curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json
    tt model stop changh95/GR00T-N1.5-3B-p150
    ```

    - `POST /predict`: `images` (`{"ego_view": <base64 PNG/JPEG>}` — the one GR1 camera; any size with sides in 64–4096 px, the server applies the checkpoint's own eval chain: 0.95 centre crop → bilinear 224×224 → Eagle2.5 normalisation), `state` (`{group: [[floats]]}`: `left_arm` 7, `right_arm` 7, `left_hand` 6, `right_hand` 6 — raw joint values, one time step), `instruction` (task text, ≤ 102 BPE tokens); optional `embodiment` (`gr1`, the only layout in this image), `seed` (initial flow-matching noise; default 0 = the deployed policy's seed → deterministic) or `noise` (an explicit `[16][32]` initial noise), `state_dtype` (`float64` default, as the GR1 dataset), `return_normalized` (also return the model-space chunk).
    - `GET /health`, `GET /info` (the full input/output contract, weights/tree provenance, warm-up latency and fidelity), `GET /demo` (the shipped demo request and its fp32 reference actions).

    ### Response

    ```json
    {"actions": {"left_arm":   [[0.0069, 0.2107, 0.0483, -1.8136, -0.1264, -0.0548, 0.0147], "... 16 steps"],
                 "right_arm":  [[-0.0177, -0.1731, 0.3702, -1.7429, 0.4968, -0.0567, 0.9849], "..."],
                 "left_hand":  [[0.0081, 0.0155, 0.0033, 0.0064, 0.0169, 0.0438], "..."],
                 "right_hand": [[-1.4880, -1.4915, -1.4781, -1.4810, -2.9609, 2.9904], "..."]},
     "action_horizon": 16, "action_keys": ["left_arm", "right_arm", "left_hand", "right_hand"], "action_dims": {"left_arm": 7, "right_arm": 7, "left_hand": 6, "right_hand": 6},
     "normalized": false, "embodiment": "gr1", "embodiment_id": 24, "version": "n15", "model": "GR00T-N1.5-3B-p150", "layout": "gr1",
     "images": {"ego_view": {"frames": 1, "received_hw": [256, 256]}}, "seq_len": 296, "prompt_tokens": 14, "state_dtype": "float64",
     "noise_source": "client", "seed": null,
     "timing_ms": {"decode": 1.5, "encode": 2.6, "device": 43.0, "decode_actions": 0.1, "total": 47.2}}
    ```

    - `actions` are **physical, un-normalised** joint targets (radians) in the GR1 arms-and-hands action space of the checkpoint's `experiment_cfg/metadata.json` statistics, 16 steps × [7, 7, 6, 6] dims — what `Gr00tPolicy.get_action` returns for this checkpoint; the values above are abridged from the shipped demo request (`GET /demo`) with its golden initial noise. `seq_len` is the LLM prompt length the request produced (296 for the demo; padded to 384 on device). `timing_ms.device` covers input upload, the four trace replays and the action readback; `total` is the whole handler.

    ### Demo

    | Input (`media/demo_ego_view.png`, GR1 sim `robot_sim.PickNPlace` traj 0 / step 100, 256×256) | Instruction | p150a actions vs the fp32 reference (`media/demo_actions.png`) |
    |:---:|:---|:---:|
    | ![](media/demo_ego_view.png) | `pick the pear from the counter and place it in the plate` | ![](media/demo_actions.png) |

    ### Accuracy and speed

    | Metric | Value |
    |---|---:|
    | Action chunk vs the fp32 reference (`Gr00tPolicy`, same observation, same initial noise) | PCC `left_arm` 0.99995 · `right_arm` 0.99997 · `right_hand` 0.99999 (gates 0.9999; max\|d\| 0.021 / 0.017 / 0.029 rad) · `left_hand` max\|d\| 0.018 rad ≤ 0.046 (PCC 0.943 reported only: the golden chunk is near-constant there, spread 0.1 rad — NVIDIA's own bf16 GPU path scores 0.952) · normalised chunk `action_pred_normalized` 0.99998 (max\|d\| 0.014 ≤ 0.027) |
    | Intermediate taps vs the fp32 reference (untraced path from pixels, 57 gated taps + determinism row) | **58/58 rows pass**; tightest: `vit_block_first` 0.99994 (gate 0.9999), `backbone_features` 0.99586 (0.995), `vl_self_attention_out` 0.99119 (0.991), `vit_post_ln` 0.99506 (0.993) |
    | Traced == untraced, determinism, stale state | traced `action_pred` / `backbone_features` / hoisted K,V bit-equal to the untraced run; 5 trace replays bit-identical; mutated inputs (`n_text` > L_max, other shapes) refused before any device write; A→B→A→B alternating observations on one captured model pass |
    | Inference on p150a (port benchmark, warm, batch 1, one 224² image, LLM S 296→384, median of 50; `benchmarks/results/e2e_stage1_n15.json`) | **43.4 ms end-to-end** (p90 44.1) = host encode 1.5 + upload 3.8 + 4 traces 37.8 (vision 8.7 · LLM 8.7 · adapter 4.5 · denoise 16.0) + readback 0.1 + decode 0.2 |
    | Served over HTTP by this image (`tt-model serve`, warm, batch 1, the demo request, 10 warm-ups + 50 timed; `scripts/bench_http.py`) | **42.4 ms device · 45.4 ms server-side** (medians; p90 43.2 / 49.2; min 41.7 / 44.5) = image decode 1.1 + host encode 1.6 + device 42.4 (upload, 4 trace replays, readback) + un-normalise 0.1; 47.6 ms client wall on the same host (p90 51.8). Under a heavily loaded host (two concurrent tt-metal builds, load average 16–110) the same loop read 44.3 / 44.5 ms device and 53.6 / 58.1 ms server-side medians with p90s of 55 / 91–97 ms — the host-side stages, not the device, absorb the contention. The served actions are bit-identical to the port's device test and to the host run |
    | Same policy on an RTX 5090 (same host, NVIDIA's unmodified `Gr00tPolicy`, eager PyTorch bf16 autocast + flash-attn 2, batch 1) | `get_action` e2e **58.0 ms** (preprocessing 2.2, model 55.4 = backbone 20.8 + action head 34.6) → the p150a is 1.34× faster end-to-end (43.4 vs 58.0) and 1.33× on the model (41.7 device vs 55.4); NVIDIA's N1.5 README quotes 47.9 ms on an H100 (backbone 23.2 + head 4 × 6.2) |

    ### Caveats

    - **Stage-1 path.** Vision / LLM / adapter / denoise are TTNN ops captured in four Metal traces; DiT block matmul weights are bfp8_b, everything else bf16 (`mixed_dit`). The persistent-megakernel denoise (Stage 2) is at rung K1 — its weight-streaming kernel measures 464 GB/s bf16 / 414 GB/s bfp8 (90.6 % of DRAM peak) on the full DiT block set — and is **not** in this image.
    - **One embodiment, one layout.** The image is built for GR1 arms-and-hands (`gr1`, embodiment slot 24, one `ego_view` camera, ≤ 102 text tokens → LLM sequence padded to 384); other embodiments, cameras or longer instructions are refused with 400, nothing is truncated. Batch 1; requests are serialised on one lock.
    - **Noise is seeded** (seed 0, as the deployed policy); pass `seed` or `noise` for another draw. The reference noise of the shipped demo is not seed-reproducible, so the smoke test sends it as `noise`. Outputs are the base checkpoint's actions for the GR1 sim data config — a real robot needs NVIDIA's post-training.
    - Not an OpenAI-compatible API; `GET /v1/models` is a stub so the tt-model ready card does not 404.
    - Validated on tt-metal main `668c2907575` (`v0.79.0-dev20260914-1`), single p150a only, with this exact image (`tt-model serve` → smoke test → 50-request benchmark → `tt-model stop`, three boots). **First boot is cold**: the checkpoint is converted into the port's device layout under `~/.cache/tt-model/gr00t-n1.5-3b-p150/weights` (6.4 GB, 9 s) and the kernels of the four traces are JIT-compiled into `…/cache` (411 MB, ~87 s incl. trace capture) — `tt-model serve` reported READY after 1 min 43 s; the next boots reused both caches and were ready in 20 s (loaded host) and 9.7 s (quiet host). Every boot re-checks the warm-up actions against the shipped fp32 reference (PCC 0.999982 here) and refuses to serve below 0.999.
    - GPU comparison: the RTX 5090 row is NVIDIA's unmodified `Gr00tPolicy` path (`Isaac-GR00T` `n1.5-release`, torch 2.7.1+cu128, transformers 4.51.3, flash-attn 2.8.3), best median of 3 × (20 warm-ups + 20 timed calls), GPU idle, no `torch.compile` / TensorRT; the p150a rows are the Stage-1 trace path incl. upload and readback. p150a power was not measured, so no efficiency comparison is made. Full table: [`GPU_COMPARISON.md`](GPU_COMPARISON.md).

    ### Licensing

    - Weights: [nvidia/GR00T-N1.5-3B](https://huggingface.co/nvidia/GR00T-N1.5-3B), **NVIDIA License** — use limited to research or evaluation (non-commercial), redistribution only under the same licence (copy in [`LICENSE`](LICENSE)). Not redistributed here; `tt-model serve` fetches the pinned snapshot into your HF cache.
    - Tokenizer files under `code/gr00t_p150/assets/tokenizer/n15/` (Qwen2 BPE + Eagle2.5 special tokens) are copied from [NVIDIA/Isaac-GR00T `n1.5-release`](https://github.com/NVIDIA/Isaac-GR00T/tree/n1.5-release) (Apache-2.0).
    - Port and serving code (`code/models/experimental/gr00t`, `code/gr00t_p150`, `code/scripts`): Apache-2.0 (SPDX headers), © 2026 Tenstorrent; as a derivative work of the weights it is distributed here under the same non-commercial use limitation (NVIDIA License §3.2). tt-metal / tt-nn: Apache-2.0.
    - Demo frame and state (`media/`): one step of the GR1 `robot_sim.PickNPlace` demo data shipped with Isaac-GR00T.