Hugging Face's logo

changh95
/
frnet-p150

Image Segmentation
autonomous-driving
autoware
blackhole
frnet
lidar
p150
point-cloud
range-view
semantic-segmentation
tenstorrent
tt-dit-server
tt-metal
tt-model-cache
tt-model-catalog
tt-model-container
tt-nn
ttnn
Model card Files
xet
 Community
frnet-p150 / SERVING.md
changh95's picture
changh95
tt-model push frnet-p150 (container)
fe51a63 verified
preview code
|
Raw History Blame
18.2 kB

Serving FRNet (Autoware lidar_frnet) on Blackhole with tt-model-manager

This repo is the authoring source of the tt-model container package changh95/frnet-p150 (kind: tt-dit-server, schema 5.1): tt-model.yaml is the manifest, code/ the port, and code/tt_frnet/server/app.py the ASGI app uvicorn runs. Weights are a pinned pointer, never baked into the image. The device open, the decoders, the Python-API contract and the HTTP contract come from code/tt_frnet/ttaw/, the shared package of the Autoware ports vendored into this repo (ttaw/VENDORED.json records its version and file hashes; it is not edited here).

item value
tt-metal tree /home/ubuntu/experiments/tt-models/tt-metal (main 44d66500520, v0.80.0-dev20261006-78-g44d66500520, + patches/tt-metal-eth-dispatch.patch; torch 2.11.0+cpu)
weights AutowareFoundation/lidar_frnet @ a53a1c11b0b28d31fd50a03abef5db0e830a5b3c (tag v2.0; files frnet_*.onnx, ml_package_frnet_*.param.yaml, deploy_metadata.yaml; 85 MB (frnet_ot128.onnx + frnet_qt128.onnx, 42.4 MB each); Apache-2.0; public, ungated)
Autoware reference perception/autoware_lidar_frnet @ autoware_universe 9ceaccf026c31ffc5319bc9eeb4bd7bede0af3fd
app tt_frnet.server.app:app (uvicorn; the ASGI lifespan does weights -> device -> graph -> trace capture)
device recipe ttnn.open_device(device_id, dispatch_core_config=DispatchCoreConfig(ETH), l1_small_size=65536, trace_region_size=256 MiB, num_command_queues=1) (DEVICE_DEFAULTS in code/tt_frnet/device.py; the trace uses 8.7 MiB (ot128) / 5.6 MiB (qt128))
hardware one Blackhole p150 (hardware: p150, mesh_device: P150, TT_MESH_SHAPE=1x1), 12×10 compute grid
serve profiles ot128 (default; Hesai OT128, FRNET_VARIANT=ot128) and qt128 (Hesai QT128); one variant per process. Both pin FRNET_DISPATCH=eth, FRNET_NUM_CQS=1 and the precision knobs FRNET_POINT_TERMS=2, FRNET_MAP_DTYPE=float32, FRNET_CONV_TERMS=1

1. Run on the HOST (hardware validation, no Docker)

The tt-metal python_env has ttnn and torch; put the HTTP stack in a side directory instead of installing into it. On the shared workspace box every command that opens the chip goes through the device lock (bin/devrun).

ROOT=/home/ubuntu/experiments/tt-models
source $ROOT/bin/tt-env.sh                     # TT_METAL_HOME, PYTHONPATH, tt-metal python_env
mkdir -p /tmp/frnet-http && uv pip install --python $TT_METAL_HOME/python_env/bin/python \
    --target /tmp/frnet-http fastapi uvicorn onnx
cd $ROOT/bundles/frnet-p150
export PYTHONPATH=$PWD/code:$TT_METAL_HOME:$TT_METAL_HOME/ttnn:/tmp/frnet-http
export HF_MODEL=AutowareFoundation/lidar_frnet TT_MODEL_WEIGHTS_REVISION=a53a1c11b0b28d31fd50a03abef5db0e830a5b3c TT_MESH_SHAPE=1x1

# host tests (no device; the device tests are skipped; ttnn is the fake of common/tests/host: a real ttnn host
# tensor opens the chip, so only devrun jobs may build one)
PYTHONPATH=$ROOT/research/packaging/scripts:$PYTHONPATH TT_VISIBLE_DEVICES=none \
    python -m pytest -q -p no:cacheprovider -p fake_ttnn_plugin code/tt_frnet/tests
# device tests and the server, under the lock
$ROOT/bin/devrun -t 1800 -- python -m pytest -q -s code/tt_frnet/tests/test_pcc_device.py code/tt_frnet/tests/test_e2e_device.py
# (FRNET_VARIANT=qt128 for the second profile; the smoke test posts the shipped sample of the served variant)
$ROOT/bin/devrun -t 1800 -- bash -c 'python -m uvicorn --host 127.0.0.1 --port 20000 --lifespan on tt_frnet.server.app:app & \
    S=$!; python code/tt_frnet/server/smoke_test.py --url http://127.0.0.1:20000 --wait 1200; R=$?; kill -TERM $S; wait $S; exit $R'

Boot log landmarks (they drive tt-model serve's progress view, boot_progress.py TT_DIT_PHASES): Loading weights -> Opening device -> Warming up: capturing trace ... -> Warmup complete -> uvicorn Application startup complete. A cold JIT cache compiles kernels for minutes; later boots take seconds. Startup failures raise and uvicorn exits non-zero (no CPU fallback). SIGTERM: the lifespan releases the traces and closes the chip (within tt-model stop's 120 s budget).

Offline override (no Hub): FRNET_WEIGHTS_DIR=<dir holding frnet_*.onnx, ml_package_frnet_*.param.yaml, deploy_metadata.yaml>.

2. Package, serve, push (Docker)

Docker 29 on this box is rootful (/var/lib/docker), and the user's session may not carry the docker group: wrap every docker-using command in sg docker -c "..." (no docker-env.sh is needed here; the reference bundles' docker-env.sh was the authors' rootless-Docker box). Run every tt-model command from this directory (source.tt_metal and extra_code[].root: code are CWD-relative) and point --out outside it.

ROOT=/home/ubuntu/experiments/tt-models; cd $ROOT/bundles/frnet-p150
TTM=$HOME/.local/share/uv/tools/tt-model/bin/python

# offline validation (seconds; no docker, no device) -- must print VALID
$TTM -c "from tt_kernel.container_manifest import load_container_manifest as L; m=L('tt-model.yaml', check_sources=True); p=m.resolve_profile(); print('VALID', m.name, m.kind, p.hardware, p.mesh_device, m.weights_ref)"

# build (one at a time on the box: the absolute lock file): ~35 min cold, ~12 min after a code/ change (the C++
# build re-runs, ccache-warm, whenever code/ changes), ~1 min for a manifest- or lock-only change
flock $ROOT/.package.lock sg docker -c "tt-model package --container tt-model.yaml --out $ROOT/build"   # log: ~/.cache/tt-model/build/frnet-p150.log
# provenance: the base-image digests this build resolved (tt-model's FROM tags float) -> build_info.json (published)
sg docker -c "python3 $ROOT/research/packaging/scripts/record_build_info.py --staged $ROOT/build/frnet-p150 --bundle ."

# serve + smoke + stop, all inside ONE device-lock window, once per serve profile (`tt-model profiles
# $ROOT/build/frnet-p150/tt_kernel_manifest.json` lists them; serve returns once READY and leaves the container
# running; the script's trap saves the container log and always runs `tt-model stop`, also when devrun's timeout
# fires: -k 150 > the 120 s grace). No token reaches the public weights download. Evidence: logs/smoke/ (--log-dir).
for prof in ot128 qt128; do
  $ROOT/bin/devrun -t 3600 -k 150 -- env -u HF_TOKEN -u HUGGING_FACE_HUB_TOKEN \
      sg docker -c "bash code/scripts/container_smoke.sh $ROOT/build/frnet-p150 20000 --profile $prof"
done

# publish: overlay the hand-written docs, the pip project and the build record onto the staged dir, then push
# (HF_TOKEN only from the environment)
rsync -a README.md SERVING.md OPT_BASELINE.md OPT_REPORT.md VERIFICATION_*.md tt-model.yaml pyproject.toml build_info.json \
    examples media patches $ROOT/build/frnet-p150/
cp code/PYTHON.md $ROOT/build/frnet-p150/PYTHON.md 2>/dev/null || true
tt-model push $ROOT/build/frnet-p150 --publish

The smoke test fails unless /info reports ETH dispatch, the 12x10 grid and what the staged package pins for the profile (FRNET_DISPATCH, FRNET_NUM_CQS, FRNET_VARIANT, the weights revision), and it compares the served output of the shipped sample of the profile (samples/synthetic_<variant>.npz) with its stored CPU-reference output (samples/synthetic_<variant>.<variant>.reference.json; >= 98.5 % label and filter agreement).

serve publishes the first free port from 20000 and exports into the container: HF_MODEL=AutowareFoundation/lidar_frnet, MESH_DEVICE=P150, TT_MESH_SHAPE=1x1, TT_MODEL_WEIGHTS_REVISION=a53a1c11b0b28d31fd50a03abef5db0e830a5b3c, then serve.env. The HF cache is mounted at /hf (rw), the JIT cache at /cache (host ~/.cache/tt-model/frnet-p150/cache), and the container sees only its own chip (--device /dev/tenstorrent/<n>). tt-cli users: tt serve changh95/frnet-p150 / tt model stop changh95/frnet-p150 (point tt at this tt-model with tt config set tools.override.tt-model ~/.local/bin/tt-model or TT_TOOL_BIN_TT_MODEL).

What push does to the repo: code/ and image/ on the Hub become exactly the staged trees (extra_code.paths + models/common/lightweightmodule.py); every top-level file of the staged dir is uploaded (the overlaid README replaces the generated card; pyproject.toml and build_info.json arrive the same way, never through code/); top-level files already on the Hub and not in the staged dir are kept.

3. The request / response contract

route purpose
GET /health {"status": "ok" | "starting" | "error", "model", "device": {"dispatch", "grid", "cores", "device_id"}, "error"}, always 200; ok only after warm-up
GET /v1/health same as /health (tt-model's hint for non-chat packages points here)
GET /info model, task, io, Autoware package + commit, weights {repo, tag, revision, path, license}, device (dispatch, grid), input limits (point formats, fields, camera order, calibration presets), labels, variant, warm variants, runtime params + defaults, compile params, warm-up and boot times
GET /v1/models {"object": "list", "data": [{"id": "AutowareFoundation/lidar_frnet", "object": "model", "owned_by": "changh95"}]} -- a stub so OpenAI-shaped probes do not 404; NOT a chat API
POST /predict one frame -> the model output (below)

3.1 Request (application/json; unknown fields are a 422)

field type meaning
points object {"format": "bin" | "npy" | "npz" | "pcd" | "list", "data": <base64 file bytes>, "fields": [...], "dtype", "key", "frame_id"}. bin = raw little-endian float32 rows, N × len(fields) (default fields x, y, z, intensity); npy / npz = (N, C) float or structured arrays; pcd = PCD v0.7 ascii / binary / binary_compressed; list = "values": [[x, y, z, i], ...] (small clouds). ONE sweep in the sensor frame (x forward, y left, z up, metres), raw intensity (0-255 for uint8 drivers), not motion-compensated: Autoware's ~/input/pointcloud. Rows with a non-finite x / y / z are dropped
calibration object {"T_base_link_from_sensor": <transform>} (the static transform the node looks up once for its ego crop box) or {"preset": "<name>"} (presets in /info: synthetic_ot128, synthetic_qt128, the shipped samples' transforms); optional "ego_crop_box": [min_x, min_y, min_z, max_x, max_y, max_z] in base_link (default Autoware's (-1.53, -1.15, -0.1) .. (6.1, 1.15, 3.1)) or null to disable the box (then no transform is needed). Required while the box is on: without it the node would skip every frame, here it is a 400
params object per-request knobs (host post-processing only; they change filter_keep, never the classes): class_probability_threshold (float in [0, 1], default 0.05), filter_classes (comma-separated class names, "" = none, default "drivable_surface", at most 16)
output_format str json (default) or npz (adds lossless base64 NPZ arrays)
sweeps, images, stream, inputs – fields of the collection's common envelope that FRNet does not take: a request with any of them is a 400

Transforms (T_<to>_from_<from>): a 4×4 (or 3×4) row-major matrix, {"translation": [x, y, z], "rotation_wxyz": [w, x, y, z]} (or rotation_xyzw), or {"x", "y", "z", "roll", "pitch", "yaw"} (Autoware sensor_kit_calibration.yaml, tf2 RPY). Intrinsics: 3×3 K, 3×4 P, or {"fx", "fy", "cx", "cy"}. Base64 may be standard or URL-safe, wrapped, with or without a data: prefix.

3.2 Response (200)

The shipped OT128 sample (synthetic_ot128.npz with the preset synthetic_ot128), abridged (data fields cut):

{"model": "frnet-p150", "frame_id": "sensor", "num_points": 52613, "num_points_raw": 52613,
 "labels": {"format": "npz", "key": "labels", "dtype": "uint8", "shape": [52613], "data": "..."},
 "scores": {"format": "npz", "key": "scores", "dtype": "float32", "shape": [52613], "data": "..."},
 "points": {"format": "npz", "key": "points", "dtype": "float32", "shape": [52613, 4], "data": "..."},
 "filter_keep": {"format": "npz", "key": "filter_keep", "dtype": "bool", "shape": [52613], "data": "..."},
 "input_index": {"format": "npz", "key": "input_index", "dtype": "int32", "shape": [52613], "data": "..."},
 "class_names": ["drivable_surface", "other_flat_surface", "sidewalk", "manmade", "vegetation", "car", "..."],
 "class_counts": {"drivable_surface": 2913, "other_flat_surface": 20118, "sidewalk": 5, "manmade": 22898, "vegetation": 6526, "car": 13, "semi_trailer": 140, "...": 0},
 "meta": {"variant": "ot128", "num_points_input": 53411, "cropped_by_ego_box": 798, "num_points_interpolated": 0, "num_unique_coors": 52613,
          "ego_crop": {"enabled": true, "bounds": [-1.53, -1.15, -0.1, 6.1, 1.15, 3.1], "reference_frame": "base_link", "...": "..."},
          "plan": {"n": 52613, "m": 52613, "n_cap": 160000, "m_cap": 60000, "...": "..."},
          "filter": {"classes": ["drivable_surface"], "class_probability_threshold": 0.05, "kept": 44747}, "...": "..."},
 "timing_ms": {"decode": 24.711, "preprocess": 47.664, "device": 229.164, "postprocess": 5.018, "model_call": 281.879, "total": 310.689}}
  • Output point order (the node's): the input points that survive the ego crop box, in input order (after non-finite rows are dropped), then the interpolated points Autoware appends; the first num_points_raw rows are raw points and input_index maps them back to the decoded input rows. The interpolated points are published, as Autoware does.
  • labels is Autoware's class_id, scores its probability (the softmax probability of that class), points the x, y, z, intensity of every output point in the sensor frame, filter_keep the membership of Autoware's ~/output/pointcloud/filtered cloud.
  • An out-of-profile frame (input points, points after interpolation or unique cells outside the variant's ranges: ot128 5,000-160,000 points, qt128 5,000-120,000, both 3,000-60,000 cells) answers 200 with num_points: 0 and the node's message in meta.skipped: the node publishes nothing for such a frame. Nothing runs on the device.
  • meta also has the crop and network-input counts (num_points_input, cropped_by_ego_box, num_points_interpolated, num_unique_coors, ...), the device plan sizes and the applied filter.

timing_ms: decode (base64 + parsing), preprocess, device (staging + H2D + trace + D2H + un-permute), postprocess, model_call, total (server side, after the body arrived).

3.3 Errors

400 undecodable / malformed input (bad base64, wrong row size, no calibration while the ego crop box is on, unknown or out-of-range params such as an unknown filter class, sweeps / images / stream / inputs, no points), 422 schema violation (unknown field, wrong type), 503 while starting (or when the boot failed), 500 inference failed: <ExceptionType>: <message> if the device call raises.

3.4 Environment the app reads (lifespan only, never at import)

var set by meaning / default
HF_MODEL launcher (weights.repo) weights repo id; default AutowareFoundation/lidar_frnet
TT_MODEL_WEIGHTS_REVISION launcher (weights.revision) pinned commit; TT_WEIGHTS_REVISION (serve.env) is the same for older tt-model
FRNET_WEIGHTS_DIR you (offline) local weights directory; overrides the Hub
TT_MESH_SHAPE launcher (runtime.mesh_shape_env) 1x1; anything else -> RuntimeError at startup
TT_DEVICE_ID you chip to open, default 0
FRNET_DISPATCH serve.env eth (default) | worker (A/B only) | auto (ETH if the patch is present)
FRNET_NUM_CQS serve.env 1
FRNET_VARIANT serve.env / profile ot128 (profile ot128) or qt128 (profile qt128)
FRNET_POINT_TERMS, FRNET_MAP_DTYPE, FRNET_CONV_TERMS serve.env the precision knobs: 2, float32, 1 (the validated configuration; other values are optimization-phase A/B switches that fail the frozen accuracy gates)
FRNET_WARMUP you JSON list of warm-up variants, default or none
FRNET_TRACE_REGION, FRNET_L1_SMALL, FRNET_WORKER_L1_SIZE you device-open overrides (validated values: DEVICE_DEFAULTS in device.py)
FRNET_MAX_BODY_MB you request size guard, default 256
TT_METAL_VISIBLE_DEVICES, MESH_DEVICE serve.env / launcher 0, P150 (informational)

Server-side pipeline: JSON -> tt_frnet.io decoders -> FRNet.__call__ under one lock: host pre-processing ported from perception/autoware_lidar_frnet (ego crop, float32 projection, fp16 interpolation grid, unique cells, profile checks, the device plan: sorted points, CSR offsets, gather tables, 8-d features) -> 9 H2D copies into persistent device inputs -> execute_trace (point MLP, K1 segment max, frustum CNN, point head, softmax) -> one D2H of the fp32 probabilities -> un-permute -> host post-processing (first-max argmax, probability, class filter) -> Output.to_dict().

4. Caveats

  • Batch 1, one sweep per request; concurrent clients queue on the lock.
  • Fixed device shapes per profile: N_cap 160,000 (ot128) / 120,000 (qt128) points and M_cap 60,000 cells; every point-wise op runs on all N_cap rows, so the device time hardly depends on the frame. Frames outside Autoware's profile ranges are skipped as the node skips them (200, empty output, meta.skipped).
  • One variant per process: serve --profile qt128 for the Hesai QT128 network. The profiles share the image.
  • Warm-up captures every trace variant inside the lifespan, so READY means warm; the first cold boot pays the ttnn JIT.
  • Weights are pinned by sha in weights.revision, exported by the launcher and repeated in serve.env; snapshot_download(..., revision=<sha>) is a cache hit after serve's pre-download and falls back to local_files_only=True if the Hub is unreachable.
  • The runtime image has no host C/C++ compiler: device kernels JIT-compile (sfpi ships in the image), host-side C helpers must have a numpy fallback.
  • tt-model curl and the ready card's /v1/models hint are OpenAI-shaped and are not this API; use the routes above.