SERVING.md

# Serving FRNet (Autoware lidar_frnet) on Blackhole with tt-model-manager

This repo is the authoring source of the **tt-model container package** `changh95/frnet-p150`
(`kind: tt-dit-server`, schema 5.1): `tt-model.yaml` is the manifest, `code/` the port, and
`code/tt_frnet/server/app.py` the ASGI app uvicorn runs. Weights are a pinned pointer, never baked into the image.
The device open, the decoders, the Python-API contract and the HTTP contract come from `code/tt_frnet/ttaw/`, the
shared package of the Autoware ports vendored into this repo (`ttaw/VENDORED.json` records its version and file
hashes; it is not edited here).

| item | value |
|---|---|
| tt-metal tree | `/home/ubuntu/experiments/tt-models/tt-metal` (main `44d66500520`, `v0.80.0-dev20261006-78-g44d66500520`, + `patches/tt-metal-eth-dispatch.patch`; torch 2.11.0+cpu) |
| weights | `AutowareFoundation/lidar_frnet` @ `a53a1c11b0b28d31fd50a03abef5db0e830a5b3c` (tag `v2.0`; files `frnet_*.onnx, ml_package_frnet_*.param.yaml, deploy_metadata.yaml`; 85 MB (frnet_ot128.onnx + frnet_qt128.onnx, 42.4 MB each); Apache-2.0; public, ungated) |
| Autoware reference | `perception/autoware_lidar_frnet` @ autoware_universe `9ceaccf026c31ffc5319bc9eeb4bd7bede0af3fd` |
| app | `tt_frnet.server.app:app` (uvicorn; the ASGI lifespan does weights -> device -> graph -> trace capture) |
| device recipe | `ttnn.open_device(device_id, dispatch_core_config=DispatchCoreConfig(ETH), l1_small_size=65536, trace_region_size=256 MiB, num_command_queues=1)` (`DEVICE_DEFAULTS` in `code/tt_frnet/device.py`; the trace uses 8.7 MiB (ot128) / 5.6 MiB (qt128)) |
| hardware | one Blackhole p150 (`hardware: p150`, `mesh_device: P150`, `TT_MESH_SHAPE=1x1`), 12×10 compute grid |
| serve profiles | `ot128` (default; Hesai OT128, `FRNET_VARIANT=ot128`) and `qt128` (Hesai QT128); one variant per process. Both pin `FRNET_DISPATCH=eth`, `FRNET_NUM_CQS=1` and the precision knobs `FRNET_POINT_TERMS=2`, `FRNET_MAP_DTYPE=float32`, `FRNET_CONV_TERMS=1` |

## 1. Run on the HOST (hardware validation, no Docker)

The tt-metal `python_env` has ttnn and torch; put the HTTP stack in a side directory instead of installing into it.
On the shared workspace box every command that opens the chip goes through the device lock (`bin/devrun`).

```bash
ROOT=/home/ubuntu/experiments/tt-models
source $ROOT/bin/tt-env.sh                     # TT_METAL_HOME, PYTHONPATH, tt-metal python_env
mkdir -p /tmp/frnet-http && uv pip install --python $TT_METAL_HOME/python_env/bin/python \
    --target /tmp/frnet-http fastapi uvicorn onnx
cd $ROOT/bundles/frnet-p150
export PYTHONPATH=$PWD/code:$TT_METAL_HOME:$TT_METAL_HOME/ttnn:/tmp/frnet-http
export HF_MODEL=AutowareFoundation/lidar_frnet TT_MODEL_WEIGHTS_REVISION=a53a1c11b0b28d31fd50a03abef5db0e830a5b3c TT_MESH_SHAPE=1x1

# host tests (no device; the device tests are skipped; ttnn is the fake of common/tests/host: a real ttnn host
# tensor opens the chip, so only devrun jobs may build one)
PYTHONPATH=$ROOT/research/packaging/scripts:$PYTHONPATH TT_VISIBLE_DEVICES=none \
    python -m pytest -q -p no:cacheprovider -p fake_ttnn_plugin code/tt_frnet/tests
# device tests and the server, under the lock
$ROOT/bin/devrun -t 1800 -- python -m pytest -q -s code/tt_frnet/tests/test_pcc_device.py code/tt_frnet/tests/test_e2e_device.py
# (FRNET_VARIANT=qt128 for the second profile; the smoke test posts the shipped sample of the served variant)
$ROOT/bin/devrun -t 1800 -- bash -c 'python -m uvicorn --host 127.0.0.1 --port 20000 --lifespan on tt_frnet.server.app:app & \
    S=$!; python code/tt_frnet/server/smoke_test.py --url http://127.0.0.1:20000 --wait 1200; R=$?; kill -TERM $S; wait $S; exit $R'
```

Boot log landmarks (they drive `tt-model serve`'s progress view, `boot_progress.py` TT_DIT_PHASES):
`Loading weights` -> `Opening device` -> `Warming up: capturing trace ...` -> `Warmup complete` -> uvicorn
`Application startup complete`. A cold JIT cache compiles kernels for minutes; later boots take seconds. Startup
failures raise and uvicorn exits non-zero (no CPU fallback). SIGTERM: the lifespan releases the traces and closes the
chip (within `tt-model stop`'s 120 s budget).

Offline override (no Hub): `FRNET_WEIGHTS_DIR=<dir holding frnet_*.onnx, ml_package_frnet_*.param.yaml, deploy_metadata.yaml>`.

## 2. Package, serve, push (Docker)

Docker 29 on this box is **rootful** (`/var/lib/docker`), and the user's session may not carry the `docker` group:
wrap every docker-using command in `sg docker -c "..."` (no `docker-env.sh` is needed here; the reference bundles'
`docker-env.sh` was the authors' rootless-Docker box). Run every `tt-model` command **from this directory**
(`source.tt_metal` and `extra_code[].root: code` are CWD-relative) and point `--out` outside it.

```bash
ROOT=/home/ubuntu/experiments/tt-models; cd $ROOT/bundles/frnet-p150
TTM=$HOME/.local/share/uv/tools/tt-model/bin/python

# offline validation (seconds; no docker, no device) -- must print VALID
$TTM -c "from tt_kernel.container_manifest import load_container_manifest as L; m=L('tt-model.yaml', check_sources=True); p=m.resolve_profile(); print('VALID', m.name, m.kind, p.hardware, p.mesh_device, m.weights_ref)"

# build (one at a time on the box: the absolute lock file): ~35 min cold, ~12 min after a code/ change (the C++
# build re-runs, ccache-warm, whenever code/ changes), ~1 min for a manifest- or lock-only change
flock $ROOT/.package.lock sg docker -c "tt-model package --container tt-model.yaml --out $ROOT/build"   # log: ~/.cache/tt-model/build/frnet-p150.log
# provenance: the base-image digests this build resolved (tt-model's FROM tags float) -> build_info.json (published)
sg docker -c "python3 $ROOT/research/packaging/scripts/record_build_info.py --staged $ROOT/build/frnet-p150 --bundle ."

# serve + smoke + stop, all inside ONE device-lock window, once per serve profile (`tt-model profiles
# $ROOT/build/frnet-p150/tt_kernel_manifest.json` lists them; serve returns once READY and leaves the container
# running; the script's trap saves the container log and always runs `tt-model stop`, also when devrun's timeout
# fires: -k 150 > the 120 s grace). No token reaches the public weights download. Evidence: logs/smoke/ (--log-dir).
for prof in ot128 qt128; do
  $ROOT/bin/devrun -t 3600 -k 150 -- env -u HF_TOKEN -u HUGGING_FACE_HUB_TOKEN \
      sg docker -c "bash code/scripts/container_smoke.sh $ROOT/build/frnet-p150 20000 --profile $prof"
done

# publish: overlay the hand-written docs, the pip project and the build record onto the staged dir, then push
# (HF_TOKEN only from the environment)
rsync -a README.md SERVING.md OPT_BASELINE.md OPT_REPORT.md VERIFICATION_*.md tt-model.yaml pyproject.toml build_info.json \
    examples media patches $ROOT/build/frnet-p150/
cp code/PYTHON.md $ROOT/build/frnet-p150/PYTHON.md 2>/dev/null || true
tt-model push $ROOT/build/frnet-p150 --publish
```

The smoke test fails unless `/info` reports ETH dispatch, the 12x10 grid and what the staged package pins for the
profile (`FRNET_DISPATCH`, `FRNET_NUM_CQS`, `FRNET_VARIANT`, the weights revision), and it compares the
served output of the shipped sample of the profile (`samples/synthetic_<variant>.npz`) with its stored CPU-reference
output (`samples/synthetic_<variant>.<variant>.reference.json`; >= 98.5 % label and filter agreement).

`serve` publishes the first free port from 20000 and exports into the container: `HF_MODEL=AutowareFoundation/lidar_frnet`,
`MESH_DEVICE=P150`, `TT_MESH_SHAPE=1x1`, `TT_MODEL_WEIGHTS_REVISION=a53a1c11b0b28d31fd50a03abef5db0e830a5b3c`, then `serve.env`. The HF cache is
mounted at `/hf` (rw), the JIT cache at `/cache` (host `~/.cache/tt-model/frnet-p150/cache`), and the container
sees only its own chip (`--device /dev/tenstorrent/<n>`). tt-cli users: `tt serve changh95/frnet-p150` /
`tt model stop changh95/frnet-p150` (point tt at this tt-model with
`tt config set tools.override.tt-model ~/.local/bin/tt-model` or `TT_TOOL_BIN_TT_MODEL`).

What `push` does to the repo: `code/` and `image/` on the Hub become exactly the staged trees (`extra_code.paths` +
`models/common/lightweightmodule.py`); every top-level file of the staged dir is uploaded (the overlaid README
replaces the generated card; `pyproject.toml` and `build_info.json` arrive the same way, never through `code/`);
top-level files already on the Hub and not in the staged dir are kept.

## 3. The request / response contract

| route | purpose |
|---|---|
| `GET /health` | `{"status": "ok" \| "starting" \| "error", "model", "device": {"dispatch", "grid", "cores", "device_id"}, "error"}`, always 200; `ok` only after warm-up |
| `GET /v1/health` | same as `/health` (tt-model's hint for non-chat packages points here) |
| `GET /info` | model, task, io, Autoware package + commit, weights {repo, tag, revision, path, license}, device (dispatch, grid), input limits (point formats, fields, camera order, calibration presets), labels, variant, warm variants, runtime params + defaults, compile params, warm-up and boot times |
| `GET /v1/models` | `{"object": "list", "data": [{"id": "AutowareFoundation/lidar_frnet", "object": "model", "owned_by": "changh95"}]}` -- a stub so OpenAI-shaped probes do not 404; NOT a chat API |
| `POST /predict` | one frame -> the model output (below) |

### 3.1 Request (`application/json`; unknown fields are a 422)

| field | type | meaning |
|---|---|---|
| `points` | object | `{"format": "bin" \| "npy" \| "npz" \| "pcd" \| "list", "data": <base64 file bytes>, "fields": [...], "dtype", "key", "frame_id"}`. `bin` = raw little-endian float32 rows, N × len(`fields`) (default fields `x, y, z, intensity`); `npy` / `npz` = (N, C) float or structured arrays; `pcd` = PCD v0.7 ascii / binary / binary_compressed; `list` = `"values": [[x, y, z, i], ...]` (small clouds). ONE sweep in the **sensor frame** (x forward, y left, z up, metres), raw intensity (0-255 for uint8 drivers), not motion-compensated: Autoware's `~/input/pointcloud`. Rows with a non-finite x / y / z are dropped |
| `calibration` | object | `{"T_base_link_from_sensor": <transform>}` (the static transform the node looks up once for its ego crop box) or `{"preset": "<name>"}` (presets in `/info`: `synthetic_ot128`, `synthetic_qt128`, the shipped samples' transforms); optional `"ego_crop_box"`: `[min_x, min_y, min_z, max_x, max_y, max_z]` in `base_link` (default Autoware's `(-1.53, -1.15, -0.1)` .. `(6.1, 1.15, 3.1)`) or `null` to disable the box (then no transform is needed). Required while the box is on: without it the node would skip every frame, here it is a 400 |
| `params` | object | per-request knobs (host post-processing only; they change `filter_keep`, never the classes): `class_probability_threshold` (float in [0, 1], default 0.05), `filter_classes` (comma-separated class names, `""` = none, default `"drivable_surface"`, at most 16) |
| `output_format` | str | `json` (default) or `npz` (adds lossless base64 NPZ arrays) |
| `sweeps`, `images`, `stream`, `inputs` | – | fields of the collection's common envelope that FRNet does not take: a request with any of them is a 400 |

Transforms (`T_<to>_from_<from>`): a 4×4 (or 3×4) row-major matrix, `{"translation": [x, y, z], "rotation_wxyz": [w, x, y, z]}`
(or `rotation_xyzw`), or `{"x", "y", "z", "roll", "pitch", "yaw"}` (Autoware `sensor_kit_calibration.yaml`, tf2 RPY).
Intrinsics: 3×3 K, 3×4 P, or `{"fx", "fy", "cx", "cy"}`. Base64 may be standard or URL-safe, wrapped, with or without a
`data:` prefix.

### 3.2 Response (200)

The shipped OT128 sample (`synthetic_ot128.npz` with the preset `synthetic_ot128`), abridged (`data` fields cut):

```json
{"model": "frnet-p150", "frame_id": "sensor", "num_points": 52613, "num_points_raw": 52613,
 "labels": {"format": "npz", "key": "labels", "dtype": "uint8", "shape": [52613], "data": "..."},
 "scores": {"format": "npz", "key": "scores", "dtype": "float32", "shape": [52613], "data": "..."},
 "points": {"format": "npz", "key": "points", "dtype": "float32", "shape": [52613, 4], "data": "..."},
 "filter_keep": {"format": "npz", "key": "filter_keep", "dtype": "bool", "shape": [52613], "data": "..."},
 "input_index": {"format": "npz", "key": "input_index", "dtype": "int32", "shape": [52613], "data": "..."},
 "class_names": ["drivable_surface", "other_flat_surface", "sidewalk", "manmade", "vegetation", "car", "..."],
 "class_counts": {"drivable_surface": 2913, "other_flat_surface": 20118, "sidewalk": 5, "manmade": 22898, "vegetation": 6526, "car": 13, "semi_trailer": 140, "...": 0},
 "meta": {"variant": "ot128", "num_points_input": 53411, "cropped_by_ego_box": 798, "num_points_interpolated": 0, "num_unique_coors": 52613,
          "ego_crop": {"enabled": true, "bounds": [-1.53, -1.15, -0.1, 6.1, 1.15, 3.1], "reference_frame": "base_link", "...": "..."},
          "plan": {"n": 52613, "m": 52613, "n_cap": 160000, "m_cap": 60000, "...": "..."},
          "filter": {"classes": ["drivable_surface"], "class_probability_threshold": 0.05, "kept": 44747}, "...": "..."},
 "timing_ms": {"decode": 24.711, "preprocess": 47.664, "device": 229.164, "postprocess": 5.018, "model_call": 281.879, "total": 310.689}}
```

- Output point order (the node's): the input points that survive the ego crop box, in input order (after non-finite
  rows are dropped), then the interpolated points Autoware appends; the first `num_points_raw` rows are raw points and
  `input_index` maps them back to the decoded input rows. The interpolated points are published, as Autoware does.
- `labels` is Autoware's `class_id`, `scores` its `probability` (the softmax probability of that class), `points` the
  x, y, z, intensity of every output point in the sensor frame, `filter_keep` the membership of Autoware's
  `~/output/pointcloud/filtered` cloud.
- An out-of-profile frame (input points, points after interpolation or unique cells outside the variant's ranges:
  ot128 5,000-160,000 points, qt128 5,000-120,000, both 3,000-60,000 cells) answers 200 with `num_points: 0` and the
  node's message in `meta.skipped`: the node publishes nothing for such a frame. Nothing runs on the device.
- `meta` also has the crop and network-input counts (`num_points_input`, `cropped_by_ego_box`, `num_points_interpolated`,
  `num_unique_coors`, ...), the device plan sizes and the applied filter.

`timing_ms`: `decode` (base64 + parsing), `preprocess`, `device` (staging + H2D + trace + D2H + un-permute),
`postprocess`, `model_call`, `total` (server side, after the body arrived).

### 3.3 Errors

**400** undecodable / malformed input (bad base64, wrong row size, no calibration while the ego crop box is on, unknown
or out-of-range `params` such as an unknown filter class, `sweeps` / `images` / `stream` / `inputs`, no `points`), **422** schema violation (unknown field, wrong type), **503** while starting
(or when the boot failed), **500** `inference failed: <ExceptionType>: <message>` if the device call raises.

### 3.4 Environment the app reads (lifespan only, never at import)

| var | set by | meaning / default |
|---|---|---|
| `HF_MODEL` | launcher (`weights.repo`) | weights repo id; default `AutowareFoundation/lidar_frnet` |
| `TT_MODEL_WEIGHTS_REVISION` | launcher (`weights.revision`) | pinned commit; `TT_WEIGHTS_REVISION` (`serve.env`) is the same for older tt-model |
| `FRNET_WEIGHTS_DIR` | you (offline) | local weights directory; overrides the Hub |
| `TT_MESH_SHAPE` | launcher (`runtime.mesh_shape_env`) | `1x1`; anything else -> RuntimeError at startup |
| `TT_DEVICE_ID` | you | chip to open, default 0 |
| `FRNET_DISPATCH` | `serve.env` | `eth` (default) \| `worker` (A/B only) \| `auto` (ETH if the patch is present) |
| `FRNET_NUM_CQS` | `serve.env` | `1` |
| `FRNET_VARIANT` | `serve.env` / profile | `ot128` (profile `ot128`) or `qt128` (profile `qt128`) |
| `FRNET_POINT_TERMS`, `FRNET_MAP_DTYPE`, `FRNET_CONV_TERMS` | `serve.env` | the precision knobs: `2`, `float32`, `1` (the validated configuration; other values are optimization-phase A/B switches that fail the frozen accuracy gates) |
| `FRNET_WARMUP` | you | JSON list of warm-up variants, `default` or `none` |
| `FRNET_TRACE_REGION`, `FRNET_L1_SMALL`, `FRNET_WORKER_L1_SIZE` | you | device-open overrides (validated values: `DEVICE_DEFAULTS` in `device.py`) |
| `FRNET_MAX_BODY_MB` | you | request size guard, default 256 |
| `TT_METAL_VISIBLE_DEVICES`, `MESH_DEVICE` | `serve.env` / launcher | `0`, `P150` (informational) |

Server-side pipeline: JSON -> `tt_frnet.io` decoders -> `FRNet.__call__` under one lock: host pre-processing ported
from `perception/autoware_lidar_frnet` (ego crop, float32 projection, fp16 interpolation grid, unique cells, profile
checks, the device plan: sorted points, CSR offsets, gather tables, 8-d features) -> 9 H2D copies into persistent
device inputs -> `execute_trace` (point MLP, K1 segment max, frustum CNN, point head, softmax) -> one D2H of the fp32
probabilities -> un-permute -> host post-processing (first-max argmax, probability, class filter) ->
`Output.to_dict()`.

## 4. Caveats

- Batch 1, one sweep per request; concurrent clients queue on the lock.
- Fixed device shapes per profile: `N_cap` 160,000 (ot128) / 120,000 (qt128) points and `M_cap` 60,000 cells; every
  point-wise op runs on all `N_cap` rows, so the device time hardly depends on the frame. Frames outside Autoware's
  profile ranges are skipped as the node skips them (200, empty output, `meta.skipped`).
- One variant per process: serve `--profile qt128` for the Hesai QT128 network. The profiles share the image.
- Warm-up captures every trace variant inside the lifespan, so READY means warm; the first cold boot pays the ttnn JIT.
- Weights are pinned by sha in `weights.revision`, exported by the launcher and repeated in `serve.env`;
  `snapshot_download(..., revision=<sha>)` is a cache hit after `serve`'s pre-download and falls back to
  `local_files_only=True` if the Hub is unreachable.
- The runtime image has no host C/C++ compiler: device kernels JIT-compile (sfpi ships in the image), host-side C
  helpers must have a numpy fallback.
- `tt-model curl` and the ready card's `/v1/models` hint are OpenAI-shaped and are not this API; use the routes above.