# Serving METEOR (TIER IV, AutowareFoundation/meteor) on Blackhole with tt-model-manager

This repo is the authoring source of the **tt-model container package** `changh95/meteor-p150`
(`kind: tt-dit-server`, schema 5.1): `tt-model.yaml` is the manifest, `code/` the port, and
`code/tt_meteor/server/app.py` the ASGI app uvicorn runs. Weights are a pinned pointer, never baked into the image.
The device open, the decoders, the Python-API contract and the HTTP contract come from `code/tt_meteor/ttaw/`, the
shared package of the Autoware ports vendored into this repo (`ttaw/VENDORED.json` records its version and file
hashes; it is not edited here).

| item | value |
|---|---|
| tt-metal tree | `/home/ubuntu/experiments/tt-models/tt-metal` (main `44d66500520`, `v0.80.0-dev20261006-78-g44d66500520`, + `patches/tt-metal-eth-dispatch.patch` + `patches/tt-metal-reshape-rm-sys1419.patch`; torch 2.11.0+cpu) |
| weights | `AutowareFoundation/meteor` @ `01a5f6d71df5ecbbb5853ec600825481d57b9c6b` (tag `v1.0`; files `meteor_v157c3Z.onnx, meteor_v157.param.yaml, LICENSE, SHA256SUMS`; Apache-2.0; public, ungated) |
| pre- / post-processing reference | **no Autoware package exists for METEOR**: METEOR's own runtimes define them, [tier4/METEOR](https://github.com/tier4/METEOR) @ `dc193a81e9959de57b751e269127ae9b67e3f300` (`deploy/cpp`: `realtime_main.cpp`, `decode.cpp`, `render.cpp`; `hf/onnx_smoke_test.py`) |
| app | `tt_meteor.server.app:app` (uvicorn; the ASGI lifespan does weights -> device -> graph -> trace capture) |
| device recipe | `ttnn.open_device(device_id, dispatch_core_config=DispatchCoreConfig(ETH), l1_small_size=32768, trace_region_size=256 MiB, num_command_queues=1)` (`DEVICE_DEFAULTS` in `code/tt_meteor/device.py`) |
| hardware | one Blackhole p150 (`hardware: p150`, `mesh_device: P150`, `TT_MESH_SHAPE=1x1`), 12×10 compute grid |

## 1. Run on the HOST (hardware validation, no Docker)

The tt-metal `python_env` has ttnn and torch; put the HTTP stack in a side directory instead of installing into it.
On the shared workspace box every command that opens the chip goes through the device lock (`bin/devrun`).

```bash
ROOT=/home/ubuntu/experiments/tt-models
source $ROOT/bin/tt-env.sh                     # TT_METAL_HOME, PYTHONPATH, tt-metal python_env
mkdir -p /tmp/meteor-http && uv pip install --python $TT_METAL_HOME/python_env/bin/python \
    --target /tmp/meteor-http fastapi uvicorn onnx
cd $ROOT/bundles/meteor-p150
export PYTHONPATH=$PWD/code:$TT_METAL_HOME:$TT_METAL_HOME/ttnn:/tmp/meteor-http
export HF_MODEL=AutowareFoundation/meteor TT_MODEL_WEIGHTS_REVISION=01a5f6d71df5ecbbb5853ec600825481d57b9c6b TT_MESH_SHAPE=1x1
# (on the shared workspace box every METEOR device job also needs METEOR_DEVICE_OK=1 from the orchestrator)

# host tests (no device; the device tests are skipped; ttnn is the fake of common/tests/host: a real ttnn host
# tensor opens the chip, so only devrun jobs may build one)
PYTHONPATH=$ROOT/research/packaging/scripts:$PYTHONPATH TT_VISIBLE_DEVICES=none \
    python -m pytest -q -p no:cacheprovider -p fake_ttnn_plugin code/tt_meteor/tests
# device tests and the server, under the lock
$ROOT/bin/devrun -t 2400 -- env TTAW_GATES_READONLY=1 python -m pytest -q -s code/tt_meteor/tests/test_pcc_device.py \
    code/tt_meteor/tests/test_e2e_device.py code/tt_meteor/tests/test_variants_device.py
$ROOT/bin/devrun -t 1800 -- bash -c 'python -m uvicorn --host 127.0.0.1 --port 20000 --lifespan on tt_meteor.server.app:app & \
    S=$!; python code/tt_meteor/server/smoke_test.py --url http://127.0.0.1:20000 --wait 1200; R=$?; kill -TERM $S; wait $S; exit $R'
```

Boot log landmarks (they drive `tt-model serve`'s progress view, `boot_progress.py` TT_DIT_PHASES):
`Loading weights` -> `Opening device` -> `Warming up: capturing trace ...` -> `Warmup complete` -> uvicorn
`Application startup complete`. The warm-up runs one eager frame of the whole graph and captures it as one trace:
about 45-50 s with a warm JIT cache (most of it the eager frame of 4,678 programs); a cold JIT cache compiles kernels
for minutes (the first container boot on an empty JIT cache was ready after 799 s, 2026-10-10). Startup
failures raise and uvicorn exits non-zero (no CPU fallback). SIGTERM: the lifespan releases the traces and closes the
chip (within `tt-model stop`'s 120 s budget).

Offline override (no Hub): `METEOR_WEIGHTS_DIR=<dir holding meteor_v157c3Z.onnx, meteor_v157.param.yaml, LICENSE, SHA256SUMS>`.

## 2. Package, serve, push (Docker)

Docker 29 on this box is **rootful** (`/var/lib/docker`), and the user's session may not carry the `docker` group:
wrap every docker-using command in `sg docker -c "..."` (no `docker-env.sh` is needed here; the reference bundles'
`docker-env.sh` was the authors' rootless-Docker box). Run every `tt-model` command **from this directory**
(`source.tt_metal` and `extra_code[].root: code` are CWD-relative) and point `--out` outside it.

```bash
ROOT=/home/ubuntu/experiments/tt-models; cd $ROOT/bundles/meteor-p150
TTM=$HOME/.local/share/uv/tools/tt-model/bin/python

# offline validation (seconds; no docker, no device) -- must print VALID
$TTM -c "from tt_kernel.container_manifest import load_container_manifest as L; m=L('tt-model.yaml', check_sources=True); p=m.resolve_profile(); print('VALID', m.name, m.kind, p.hardware, p.mesh_device, m.weights_ref)"

# build (one at a time on the box: the absolute lock file): ~35 min cold, ~12 min after a code/ change (the C++
# build re-runs, ccache-warm, whenever code/ changes), ~1 min for a manifest- or lock-only change
flock $ROOT/.package.lock sg docker -c "tt-model package --container tt-model.yaml --out $ROOT/build"   # log: ~/.cache/tt-model/build/meteor-p150.log
# provenance: the base-image digests this build resolved (tt-model's FROM tags float) -> build_info.json (published)
sg docker -c "python3 $ROOT/research/packaging/scripts/record_build_info.py --staged $ROOT/build/meteor-p150 --bundle ."

# serve + smoke + stop, all inside ONE device-lock window, once per serve profile (`tt-model profiles
# $ROOT/build/meteor-p150/tt_kernel_manifest.json` lists them; serve returns once READY and leaves the container
# running; the script's trap saves the container log and always runs `tt-model stop`, also when devrun's timeout
# fires: -k 150 > the 120 s grace). No token reaches the public weights download. Evidence: logs/smoke/ (--log-dir).
$ROOT/bin/devrun -t 3600 -k 150 -- env -u HF_TOKEN -u HUGGING_FACE_HUB_TOKEN \
    sg docker -c "bash code/scripts/container_smoke.sh $ROOT/build/meteor-p150 20000 [--profile NAME]"

# publish: overlay the hand-written docs, the pip project and the build record onto the staged dir, then push
# (HF_TOKEN only from the environment)
rsync -a README.md SERVING.md OPT_BASELINE.md OPT_REPORT.md VERIFICATION_*.md tt-model.yaml pyproject.toml build_info.json \
    examples media patches $ROOT/build/meteor-p150/
cp PYTHON.md $ROOT/build/meteor-p150/PYTHON.md     # the top-level copy (its links are relative to the repo root)
tt-model push $ROOT/build/meteor-p150 --publish
```

The smoke test fails unless `/info` reports ETH dispatch, the 12x10 grid and what the staged package pins for the
profile (`METEOR_DISPATCH`, `METEOR_NUM_CQS`, `METEOR_VARIANT`, the weights revision), and it compares the
served output of the shipped sample (`code/tt_meteor/samples/synthetic_8cam.json`) with its stored CPU-reference
output (`synthetic_8cam.reference.json`: 3D boxes within 0.5 m and 0.05 of score, recall and precision >= 0.95; lane
map agreement >= 0.99; selected ego path ADE <= 0.3 m, FDE <= 0.6 m) and checks the expected boxes per label.

For the first release of this bundle the container check also runs the rig-switch stress (the ETH-dispatch hang
history of PORT_LOG.md section 12): `code/scripts/stress_frames.py --mode rigs --rigs shipped` for >= 200 frames
inside the image, with `tt-model serve --print`'s device, cache and env arguments (no port) plus
`METEOR_WEIGHTS_DIR=/hf/hub/models--AutowareFoundation--meteor/snapshots/<revision>` (the script reads the ONNX
directly, without the server's HF resolve; `serve` has pre-fetched the snapshot):
`python -u /opt/tt-metal/scripts/stress_frames.py --mode rigs --rigs shipped --frames 200 --jsonl <mounted dir>/stress.jsonl`.

`serve` publishes the first free port from 20000 and exports into the container: `HF_MODEL=AutowareFoundation/meteor`,
`MESH_DEVICE=P150`, `TT_MESH_SHAPE=1x1`, `TT_MODEL_WEIGHTS_REVISION=01a5f6d71df5ecbbb5853ec600825481d57b9c6b`, then `serve.env`. The HF cache is
mounted at `/hf` (rw), the JIT cache at `/cache` (host `~/.cache/tt-model/meteor-p150/cache`), and the container
sees only its own chip (`--device /dev/tenstorrent/<n>`). tt-cli users: `tt serve changh95/meteor-p150` /
`tt model stop changh95/meteor-p150` (point tt at this tt-model with
`tt config set tools.override.tt-model ~/.local/bin/tt-model` or `TT_TOOL_BIN_TT_MODEL`).

What `push` does to the repo: `code/` and `image/` on the Hub become exactly the staged trees (`extra_code.paths` +
`models/common/lightweightmodule.py`); every top-level file of the staged dir is uploaded (the overlaid README
replaces the generated card; `pyproject.toml` and `build_info.json` arrive the same way, never through `code/`);
top-level files already on the Hub and not in the staged dir are kept.

## 3. The request / response contract

| route | purpose |
|---|---|
| `GET /health` | `{"status": "ok" \| "starting" \| "error", "model", "device": {"dispatch", "grid", "cores", "device_id"}, "error"}`, always 200; `ok` only after warm-up |
| `GET /v1/health` | same as `/health` (tt-model's hint for non-chat packages points here) |
| `GET /info` | model, task, io, Autoware package + commit, weights {repo, tag, revision, path, license}, device (dispatch, grid), input limits (point formats, fields, camera order, calibration presets), labels, variant, warm variants, runtime params + defaults, compile params, warm-up and boot times |
| `GET /v1/models` | `{"object": "list", "data": [{"id": "AutowareFoundation/meteor", "object": "model", "owned_by": "changh95"}]}` -- a stub so OpenAI-shaped probes do not 404; NOT a chat API |
| `POST /predict` | one frame -> the model output (below) |

### 3.1 Request (`application/json`; unknown fields are a 422)

| field | type | meaning |
|---|---|---|
| `images` | list | **all eight cameras** `[{"camera": "CAM_FRONT_WIDE", "data": <base64 PNG / JPEG / .npy>, "intrinsics", "T_ref_from_camera"}, ...]`, any order (reordered to `CAM_FRONT_WIDE, CAM_FRONT_LEFT, CAM_FRONT_RIGHT, CAM_BACK_WIDE, CAM_BACK_LEFT, CAM_BACK_RIGHT, CAM_FRONT_NARROW, CAM_BACK_NARROW`); RGB, raw and unrectified, at least 768×432 (larger frames are resized with OpenCV INTER_AREA semantics and K is scaled per axis). An absent narrow camera is sent as an **all-zero image**: the host gives it a zero input and its donor's K and pose (`CAM_FRONT_WIDE` / `CAM_BACK_WIDE`), the trained 7-camera configuration |
| `calibration` | object | `{"frame_id", "cameras": {name: {"intrinsics", "T_ref_from_camera", "image_size"}}}` (K of the image as sent; `T_ref_from_camera` = camera optical frame -> base_link: x forward, y left, z up, origin on the road) or `{"preset": "<name>"}` (presets in `/info`: `synthetic_8cam`, the rig of the shipped sample). Inline image calibration wins |
| `ego_speed` | number | **required**: the ego speed in m/s (the graph's `v0`) |
| `stream` | object | optional: `{"id": "default", "reset": false, "timestamp_s", "T_world_from_ego"}`: METEOR's host temporal post-processing per stream id (the BEV seg fusion needs `T_world_from_ego`; yaw smoothing; plan-mode hysteresis); a new id or `reset` starts fresh. Without `stream`, every request uses the stream `default` |
| `params` | object | per-request knobs (host post-processing only; METEOR's C++ renderer defaults): 3D `det3d_threshold` (0.15), `det3d_topk` (64), `vehicle_threshold` (0.35), `vru_threshold` (0.15), `bev_nms_iou` (0.3), `bev_nms_containment` (0.6), `stationary_logit_threshold` (0.0); 2D `det2d_threshold` (0.30), `det2d_topk` (48), `det2d_hide` ("7": road paint); unknown obstacles `unk2d` (true), `unk2d_threshold` (= det2d), `ground_z` (0.0); plan `mode_hysteresis` (0.35), `straight_margin` (1.0); `seg_fuse` (true), `thin_road_edge` (true), `yaw_smoothing` (true); `heads` (false: with `output_format: "npz"`, adds the dense heads) |
| `output_format` | str | `json` (default) or `npz` (adds lossless base64 NPZ arrays) |

`points`, `sweeps` and `inputs` of the common envelope are not used by this model (a request with `points` is a 400).
`server/client.py` builds a request: `--sample <manifest>` (the shipped sample: images, preset, ego speed, stream),
or `--image NAME=PATH` (×8) + `--calib <file>` / `--calib-preset <name>` + `--ego-speed <m/s>` [+ `--stream-id`,
`--param k=v`].

Transforms (`T_<to>_from_<from>`): a 4×4 (or 3×4) row-major matrix, `{"translation": [x, y, z], "rotation_wxyz": [w, x, y, z]}`
(or `rotation_xyzw`), or `{"x", "y", "z", "roll", "pitch", "yaw"}` (Autoware `sensor_kit_calibration.yaml`, tf2 RPY).
Intrinsics: 3×3 K, 3×4 P, or `{"fx", "fy", "cx", "cy"}`. Base64 may be standard or URL-safe, wrapped, with or without a
`data:` prefix.

### 3.2 Response (200)

```json
{"model": "meteor-p150", "frame_id": "base_link", "num_detections": 8,
 "detections": [
   {"label": "VEHICLE", "label_id": 0, "score": 0.9932, "center": [-12.86, 0.095], "size": [4.399, 1.742], "yaw": -0.0687, "stationary": false, "future_mode": 0, "future": [[-11.92, -0.09], [-10.74, -0.15], ...]},
   {"label": "VEHICLE", "label_id": 0, "score": 0.9872, "center": [-7.176, -3.215], "size": [3.965, 1.731], "yaw": 0.0183, "stationary": false, ...},
   {"label": "VEHICLE", "label_id": 0, "score": 0.9624, "center": [10.84, 0.019], "size": [4.334, 1.761], "yaw": -0.0189, "stationary": false, ...},
   ...],
 "trajectory": [[0.74, 0.91], [0.85, 1.54], [0.41, 2.34], [0.31, 3.08], [0.62, 3.82], [0.3, 4.45]], "columns": ["x", "y"],
 "plan": {"mode": 0, "mode_probs": [0.233, 0.296, 0.471], "mode_logits": [...], "paths": [...], "steer": 0.0026, "accel": 0.021, "brake_prob": 0.131, "dt": 0.5},
 "detections_2d": {"CAM_FRONT_WIDE": [{"label": "car", "label_id": 1, "score": 0.5239, "box_xyxy": [412.76, 211.02, 438.5, 230.29]}, ...], ...},
 "unknown_obstacles": [],
 "traffic_light": {"state": "red", "state_id": 3, "probs": [0.363, 0.140, 0.031, 0.465]},
 "lane": {"format": "png", "key": "lane", "dtype": "uint8", "shape": [800, 500], "data": "iVBORw0KGgo..."},
 "lane_classes": ["bg", "road", "sidewalk", "crosswalk", "laneline", "stopline", "road_edge", "marking", "parking"],
 "stationary_head_healthy": true,
 "meta": {"present": [true, true, true, true, true, true, true, true], "v0": 8.0, "stream_id": "synthetic_8cam"},
 "timing_ms": {"decode": 83.95, "preprocess": 10.69, "device": 722.42, "postprocess": 86.08, "model_call": 820.65, "total": 908.67}}
```

(The shipped synthetic sample, a request after the warm-up, host server, 2026-10-10; trimmed.)

| key | content |
|---|---|
| `detections` | 3D BEV boxes sorted by score: `label` (VEHICLE / VRU), `label_id`, `score`, `center` [x, y] and `size` [length, width] in metres (base_link: x forward, y left; METEOR predicts no z, height or velocity), `yaw` (rad, CCW from +x), `stationary`, `future` (6 × [x, y] at 0.5-3 s, the agent's best mode; METEOR's renderer draws vehicle futures only), `future_mode` |
| `trajectory`, `columns` | the selected ego path, 6 × [x, y] at 0.5 s (straight preference and per-stream mode hysteresis, METEOR's `render.cpp`) |
| `plan` | `mode`, `mode_probs` (softmax of the hysteresis-adjusted logits), `mode_logits`, `paths` (3 × 6 × [x, y]), `steer` (rad), `accel` (m/s²), `brake_prob`, `dt` |
| `detections_2d` | per camera: `label` (10 classes; road paint hidden by default), `label_id`, `score`, `box_xyxy` (pixels at 768×432) |
| `unknown_obstacles` | 2D "obstacle" boxes placed on the ground plane (`unk2d`) |
| `traffic_light` | the ego-relevant light: `state` (none / green / yellow / red), `state_id`, `probs` |
| `lane`, `lane_classes` | the BEV lane map as PNG (uint8 [800, 500], 0.2 m cells, row 0 = 80 m ahead, column 0 = 50 m to the left), after the seg fusion (with a stream pose) and the road-edge thinning |
| `stationary_head_healthy`, `meta` | METEOR's stationary-head health check; the present cameras, `v0`, the stream id |
| `heads` | only with `output_format: "npz"` and `params.heads: true`: seg2d, depth, depth_mean, occupancy, risk, stationary |

The synthetic sample is a test pattern (a red light and a car 11 m ahead): its plan barely moves forward (under 1 m in 3 s,
drifting 4.5 m to the left), not meaningful driving; the stored CPU reference has the same path.

`timing_ms`: `decode` (base64 + parsing), `preprocess`, `device` (H2D + trace + D2H), `postprocess`, `model_call`,
`total` (server side, after the body arrived).

### 3.3 Errors

**400** undecodable / malformed input (bad base64, wrong row size, missing calibration, unknown or out-of-range
`params`, a missing required input), **422** schema violation (unknown field, wrong type), **503** while starting
(or when the boot failed), **500** `inference failed: <ExceptionType>: <message>` if the device call raises.

### 3.4 Environment the app reads (lifespan only, never at import)

| var | set by | meaning / default |
|---|---|---|
| `HF_MODEL` | launcher (`weights.repo`) | weights repo id; default `AutowareFoundation/meteor` |
| `TT_MODEL_WEIGHTS_REVISION` | launcher (`weights.revision`) | pinned commit; `TT_WEIGHTS_REVISION` (`serve.env`) is the same for older tt-model |
| `METEOR_WEIGHTS_DIR` | you (offline) | local weights directory; overrides the Hub |
| `TT_MESH_SHAPE` | launcher (`runtime.mesh_shape_env`) | `1x1`; anything else -> RuntimeError at startup |
| `TT_DEVICE_ID` | you | chip to open, default 0 |
| `METEOR_DISPATCH` | `serve.env` | `eth` (default) \| `worker` (A/B only) \| `auto` (ETH if the patch is present) |
| `METEOR_NUM_CQS` | `serve.env` | `1` |
| `METEOR_VARIANT` | `serve.env` / profile | `default` |
| `METEOR_WARMUP` | you | JSON list of warm-up variants, `default` or `none` |
| `METEOR_TRACE_REGION`, `METEOR_L1_SMALL`, `METEOR_WORKER_L1_SIZE` | you | device-open overrides (validated values: `DEVICE_DEFAULTS` in `device.py`) |
| `METEOR_MAX_BODY_MB` | you | request size guard, default 256 |
| `TT_METAL_VISIBLE_DEVICES`, `MESH_DEVICE` | `serve.env` / launcher | `0`, `P150` (informational) |

Server-side pipeline: JSON -> `tt_meteor.io` decoders -> `METEOR.__call__` (host pre-processing of METEOR's runtime:
INTER_AREA resize, K scaling, extrinsics inversion, the absent-camera donor rule; the lift tables of a new
calibration, cached per rig -> H2D of the eight cameras and `v0` into persistent device inputs -> `execute_trace` of
the `frame` trace -> segmented D2H -> host post-processing ported from `deploy/cpp`: 3D peaks + rotated BEV NMS,
stationary flags, agent futures, 2D boxes, unknown obstacles, the plan with hysteresis, seg fusion) under one lock
-> `Output.to_dict()`.

## 4. Caveats

- Batch 1, one frame per request; concurrent clients queue on the lock.
- Fixed shapes of the released graph: eight 768×432 camera slots in METEOR's order, a 400×250 lift grid at 0.4 m,
  an 800×500 BEV at 0.2 m; at most 3 cameras per lift cell on the rigs validated. A new calibration costs about
  0.2 s once (the lift tables), then it is cached per rig.
- The request bodies are large (eight images as base64): about 1-4 MB for PNG / JPEG at 768×432.
- Warm-up captures every trace variant inside the lifespan, so READY means warm; the first cold boot pays the ttnn JIT.
- Weights are pinned by sha in `weights.revision`, exported by the launcher and repeated in `serve.env`;
  `snapshot_download(..., revision=<sha>)` is a cache hit after `serve`'s pre-download and falls back to
  `local_files_only=True` if the Hub is unreachable.
- The runtime image has no host C/C++ compiler: device kernels JIT-compile (sfpi ships in the image), host-side C
  helpers must have a numpy fallback.
- `tt-model curl` and the ready card's `/v1/models` hint are OpenAI-shaped and are not this API; use the routes above.