PYTHON.md

# Python API: FRNet (Autoware lidar_frnet) on Blackhole

Use this API from Python code (a pipeline, a notebook, a ROS 2 node wrapper). You do not need the HTTP server: the
API and the server share the decoders, the device graph and the post-processing, so the outputs and the speed are
the same.

## Install

Install the package on top of an environment that already has `ttnn` (a tt-metal `python_env` at `44d66500520`
with `patches/tt-metal-eth-dispatch.patch`, or the tt-model container). From the root of the model repository (the
directory that holds `pyproject.toml`, `README.md` and `code/`):

```bash
pip install -e .                        # the Python API
pip install -e ".[server,test]"         # + the HTTP server and the tests
```

The pip project is the repository's top-level `pyproject.toml`; it installs the package from `code/tt_frnet` (there is no
`pyproject.toml` inside `code/`, because the container build copies `code/` over the tt-metal tree).

The package carries `tt_frnet.ttaw`, the shared code of the Autoware ports to Blackhole (device open, trace runner,
decoders, model base class, HTTP app, the K1 `segment_reduce` kernel), vendored at the version recorded in
`code/tt_frnet/ttaw/VENDORED.json` (0.20.0).

| You want to run | Extras |
|---|---|
| the Python API | none |
| the HTTP server (`tt_frnet.server.app`, see `SERVING.md`) | `server` |
| host tests (no device; device tests are skipped): `TT_VISIBLE_DEVICES=none python -m pytest -q code/tt_frnet/tests` | `server,test` |
| device tests: `python -m pytest -q -s code/tt_frnet/tests/test_pcc_device.py code/tt_frnet/tests/test_e2e_device.py` | `test` |

## Quickstart

Run from the repository root (the sample paths are relative to it):

```python
from tt_frnet import FRNet

with FRNet.from_pretrained(device_id=0) as model:                  # variant="qt128" for the Hesai QT128 profile
    out = model("code/tt_frnet/samples/synthetic_ot128.npz",       # one sweep in the sensor frame
                calibration={"preset": "synthetic_ot128"})         # its sensor -> base_link transform (ego crop box)

print(out.class_counts())                                          # points per class (raw + interpolated)
labels, scores, xyz = out.label_ids, out.scores, out.points[:, :3]
```

`examples/quickstart.py` runs this snippet and writes `quickstart.json` (the `/predict` body) and `quickstart_bev.png`
(a bird's-eye view of the p150 classes next to the stored CPU reference of the shipped sample).

## `FRNet.from_pretrained(...)`

```python
FRNet.from_pretrained(
    model_id=None,          # HF repo or a local directory with the weights files; default AutowareFoundation/lidar_frnet
    *,
    revision=None,          # default for the default repo: the validated commit a53a1c11b0b (tag v2.0)
    variant=None,           # load-time variant: "ot128" (Hesai OT128, Autoware's default) | "qt128"; default $FRNET_VARIANT or "ot128"
    device_id=None,         # chip to open; default $TT_DEVICE_ID or 0
    device=None,            # an already-opened ttnn device (tt_frnet.device.open_device); close() does not close it
    dispatch=None,          # "eth" (p150 target, 12x10 grid) | "worker" (A/B only, 11x10) | "auto"; default $FRNET_DISPATCH or "eth"
    num_command_queues=None,  # default $FRNET_NUM_CQS or 1
    weights_dir=None,       # explicit local weights directory; no Hub access
    warmup_variants="default",  # trace variants to capture now; see "Warm-up"
    verbose=False,
) -> FRNet
```

What it does: resolves the weights first, so a Hub problem never claims the chip (`weights_dir` >
`$FRNET_WEIGHTS_DIR` > a local `model_id` directory > the HF snapshot at the pinned revision, restricted to
`frnet_*.onnx, ml_package_frnet_*.param.yaml, deploy_metadata.yaml`, with an offline fallback to the cache), checks
the sha256 of the variant's ONNX against the pinned one, reads the variant's `ml_package_frnet_<variant>.param.yaml`
(FOV, frustum and interpolation grids, the 27 class names, the palette, the point / cell profiles), opens the chip
(ETH dispatch, 12×10, 1 CQ, `l1_small_size` 64 KiB, `trace_region_size` 256 MiB: `DEVICE_DEFAULTS` in
`tt_frnet/device.py`, overridable with `FRNET_*`), builds the graph, then compiles and captures the metal trace. If
ETH dispatch cannot open (tt-metal without the patch), it warns and falls back to WORKER dispatch.

`variant` picks one of the two networks of the weights repo. Each is tied to its sensor (Autoware's `sensor_model`):

| variant | sensor | FOV up / down | frustum grid (H × W) | interpolation grid | points (min / max) | unique cells (min / max) |
|---|---|---|---|---|---|---|
| `ot128` (default) | Hesai OT128 | +15° / −25° | 128 × 1024 | 128 × 4096 | 5,000 / 160,000 | 3,000 / 60,000 |
| `qt128` | Hesai QT128 | +52.6° / −52.6° | 128 × 256 | 128 × 1024 | 5,000 / 120,000 | 3,000 / 60,000 |

Precision knobs (read once at load from the environment, pinned in the package's `serve.env`; the defaults are the
validated configuration and every number of the model card uses them): `FRNET_POINT_TERMS=2` (each point-wise linear
as two-term bf16 operands, about 16 mantissa bits, fp32 accumulation), `FRNET_MAP_DTYPE=float32` (fp32 image-branch
maps, bf16 conv operands), `FRNET_CONV_TERMS=1`. Other values are A/B switches for the optimization phase; they fail
the frozen accuracy gates (`OPT_BASELINE.md`, "The precision cost of the policy").

## Warm-up

`from_pretrained` returns a warm model: it builds the graph of the variant, runs it once eagerly on an empty frame
(this first run compiles the kernels and uploads the weights) and captures it as ONE metal trace, so no call compiles
anything. There are no shape buckets: every frame is padded to the profile's fixed device shapes (`N_cap` points =
160,000 / 120,000, `M_cap` cells = 60,000), so one trace serves every input of the profile.

- Measured (2026-10-09, the shipped sample): with an empty JIT cache the load took 156.6 s for ot128 and
  151.9 s for qt128 (firmware and every kernel compiled); with that cache warm, in a new process,
  7.5 s and 6.6 s. The trace is captured during the load, so the first call costs about as
  much as the next ones (320.0 / 304.7 ms for ot128, 241.1 / 214.9 ms for qt128, first / second call, on a shared host).
- `warmup_variants="none"` (server: `FRNET_WARMUP=none`) skips the capture at load; call `model.warmup()` before the
  first frame (it is idempotent).
- The JIT cache is tt-metal's (`~/.cache/tt-metal-cache`, or `TT_METAL_CACHE`); inside the container it is
  `~/.cache/tt-model/frnet-p150/cache` of the host.

## Call: `model(points, calibration=..., **kwargs)`

| Argument | Type | Description |
|---|---|---|
| `points` (first positional) | path, `bytes` + `fmt=`, (N, C) array / tensor, structured array, `PointCloud`, JSON envelope | ONE LiDAR sweep in the **sensor frame** (x forward, y left, z up, metres), fields `x, y, z, intensity` with the raw intensity the node casts (0-255 for uint8 drivers); see "Input types" |
| `calibration` | `{"T_base_link_from_sensor": <transform>}`, `{"preset": name}` (a file of `code/tt_frnet/calib/`), optional `"ego_crop_box"` | the static sensor -> `base_link` transform the node looks up once for its ego crop box. `ego_crop_box`: `[min_x, min_y, min_z, max_x, max_y, max_z]` in `base_link` (default Autoware's `(-1.53, -1.15, -0.1)` .. `(6.1, 1.15, 3.1)`), or `None` (or all zeros) to disable the box, the node's own switch; a disabled box needs no transform |
| `class_probability_threshold` | float in [0, 1], default 0.05 | the filtered cloud keeps a point unless its probability of a filter class is >= this value (Autoware's `filter.class_probability_threshold`) |
| `filter_classes` | comma-separated class names, `""` = none, default `"drivable_surface"` | Autoware's `filter.classes` (at most 16); an unknown name is refused |

Both knobs only change `filter_keep`; the class decision (`label_ids`, `scores`) never depends on them. They are
host-only (no device shape changes), the defaults reproduce Autoware's `frnet.param.yaml`, and a bad value raises
`tt_frnet.io.InputError` (HTTP 400) before the device runs. `model.info["runtime_params"]` (and the server's `/info`)
lists them with their defaults.

### Input types

- Point clouds: a path (`.bin` raw float32 rows -- pass `fields=` if not `x, y, z, intensity`; `.npy`; `.npz`;
  `.pcd` ascii / binary / binary_compressed), raw bytes with `fmt=`, an (N, C) numpy array or torch tensor, a
  structured array, the JSON envelope of `/predict`, or `tt_frnet.io.PointCloud`. The frame is the **sensor** frame
  (Autoware's `~/input/pointcloud` before any transform), one sweep, not motion-compensated.
- Rows with a non-finite x / y / z are dropped by the decoder; `input_index` refers to the decoded rows.
- Calibration transforms: a 4×4 (or 3×4) row-major matrix, `{"translation": [x, y, z], "rotation_wxyz": [w, x, y, z]}`,
  or Autoware's `{x, y, z, roll, pitch, yaw}`. With the crop box on and no transform the call is refused (the node would
  skip every frame because its TF lookup fails); the error names both fixes.
- `sweeps`, `images`, `stream` and `inputs` are refused: FRNet takes one sweep and its calibration only.
- A frame outside the variant's profile (input points, points after interpolation, or unique cells out of the table
  above) is not an error: as the node, which logs and publishes nothing, the call returns an empty output whose
  `meta["skipped"]` holds the node's message, and nothing runs on the device.

## Output

`model(...)` returns a `tt_frnet.FRNetSegmentation` (a `ttaw.outputs.Segmentation3D`); `out.to_dict()` is exactly the
`POST /predict` body (SERVING.md section 3.2). The point order is the node's: the input points that survive the ego
crop box, in input order, then the points Autoware's interpolation appends (D4: published as Autoware does).

| field | type | meaning |
|---|---|---|
| `label_ids` | uint8 `[N]` | Autoware's `class_id` (first maximum of the 27 probabilities, as `fill_cloud_kernel`); names in `class_names` |
| `scores` | float32 `[N]` | Autoware's `probability`: the softmax probability of that class |
| `points` | float32 `[N, 4]` | x, y, z, intensity of every output point in the sensor frame (the segmentation cloud's coordinates; the interpolated points carry interpolated values) |
| `filter_keep` | bool `[N]` | membership of Autoware's `~/output/pointcloud/filtered` cloud (False = removed by the class filter) |
| `num_points_raw` | int | the first `num_points_raw` rows are input points, the rest are interpolated |
| `input_index` | int32 `[num_points_raw]` | the decoded input row of every raw output point |
| `class_names` | 27 str | `drivable_surface`, `other_flat_surface`, `sidewalk`, `manmade`, `vegetation`, `car`, `bus`, `emergency_vehicle`, `train`, `truck`, `tractor_unit`, `semi_trailer`, `construction_vehicle`, `forklift`, `kart`, `motorcycle`, `bicycle`, `pedestrian`, `personal_mobility`, `animal`, `pushable_pullable`, `traffic_cone`, `debris`, `stroller`, `other_stuff`, `noise+ghost_point`, `unknown` (Autoware's names; ids 22 / 23 were trained as stroller / debris, see Limits) |
| `meta` | dict | `variant`; `num_points_input`; `ego_crop` (the box and transform used); the network-input counts (`num_points_raw`, `num_points_interpolated`, `num_points`, `num_unique_coors`, `cropped_by_ego_box`, `crop_applied`, `max_points_per_cell`); `plan` (`n`, `m`, `n_cap`, `m_cap`); `filter` (classes, threshold, points kept); `skipped` (the node's message) for an out-of-profile frame |
| `timing_ms` | dict | `preprocess`, `device` (staging + H2D + trace + D2H + un-permute), `postprocess`, `total` of this call |

`out.class_counts()` gives the points per class (raw + interpolated); `out.to_dicts()` one row per class
(`label`, `label_id`, `points`); `len(out)` the number of output points. The node's visualization cloud is
`model.config.palette` indexed by `label_ids` (`tt_frnet.host.postprocess.visualization_rgb`).

## Lifetime and information

- `model.close()` releases the trace and the device tensors and closes the chip if the model opened it; idempotent.
  `with` calls it for you; an unclosed model is closed when Python exits.
- `model.info`: weights (repo, tag, revision, path), device (`dispatch`, `grid`, `cores`, `num_command_queues`,
  `fallback`), variant, warm variants, warm-up times, runtime parameter defaults, the variant config (`frnet`), the
  trace and graph descriptions.
- Calls from several threads are safe: the device calls are serialised by a lock. One model per process per chip.
- `tt_frnet.viz.render_bev(points, label_ids, palette=...)` and `render_compare(...)` draw bird's-eye views (PIL, no
  OpenCV).
- `model.device_probs(plan)` returns the device's `pred_probs` `(N, 27)` of a prepared frame (tests and tools; call it
  under `model._lock`).

## Speed

Warm calls, batch 1, ETH dispatch, 1 CQ, 12x10, the serve.env precision knobs (`code/scripts/bench.py`, 100
iterations; the numbers of `OPT_BASELINE.md`, 2026-10-09, on a shared host whose load moves the host stages):

| stage, p50 | ot128, shipped sample | ot128, PandaSet 019 f40 | qt128, shipped sample | qt128, PandaSet 019 f40 |
|---|---:|---:|---:|---:|
| `model(path)` end to end | 292.6 ms | 355.7 ms | 221.1 ms | 244.3 ms |
| host pre-processing (decode, crop, projection, fp16 interpolation, unique cells, device plan) | 55.3 ms | 107.2 ms | 54.2 ms | 83.5 ms |
| H2D (9 inputs, 8.6 / 6.2 MB) | 5.42 ms | 5.41 ms | 4.04 ms | 4.00 ms |
| device trace, one forward | 210.02 ms | 211.72 ms | 133.53 ms | 134.91 ms |
| D2H (fp32 `pred_probs`) | 7.65 ms | 7.43 ms | 5.80 ms | 5.55 ms |
| un-permute + host post-processing (argmax, filter) | 7.1 + 5.0 ms | 12.8 + 14.8 ms | 6.1 + 6.8 ms | 5.9 + 10.5 ms |

- Back-to-back replays take 211.64 ms (ot128) and 134.87 ms (qt128) per PandaSet frame: 4.72 and 7.41 frames/s of
  device throughput. A synchronous call gives 2.81 / 4.09 frames/s on that frame.
- The device time barely depends on the frame: every point-wise op runs on the profile's `N_cap` rows. The host
  stages grow with the cloud.
- The shipped samples are synthetic scans: ot128 53,411 points (N = M = 52,613 after the crop, no interpolated point),
  qt128 45,783 points (N = 69,553 after interpolation, M = 18,926). PandaSet 019 frame 40 is a 107,146-point Pandar64
  sweep (N = 145,961 / M = 40,691 on ot128); it is not shipped.
- `OPT_REPORT.md` ranks what the optimization phase will change (the two-term point layers, K1, the image-branch glue,
  shape buckets, the host pre-processing).

## Tests

From the repository root, with `code/` on `PYTHONPATH` (or the package installed):

- Host (no device, no ttnn call): `TT_VISIBLE_DEVICES=none python -m pytest -q code/tt_frnet/tests`. Inside the
  tt-models workspace the suite runs on its fake ttnn (`-p fake_ttnn_plugin`), which also enables `test_tt_host.py`;
  tests that need workspace-only oracles or data skip elsewhere, and FAIL inside the workspace when that data is
  missing. `test_reference_cpu.py` needs onnxruntime.
- Device (one p150): `python -m pytest -q -s code/tt_frnet/tests/test_pcc_device.py
  code/tt_frnet/tests/test_e2e_device.py` (per-module PCC against the fp32 CPU reference on replay outputs, the K1
  segment-max sites included; end-to-end agreement with ONNX Runtime on 17 OT128 / 5 QT128 frames, the shipped samples
  against their stored references, replay == eager, API == `/predict`). The public-frame gates run only inside the
  tt-models workspace, where their data lives: PandaSet, nuScenes and Autoware's sample rosbag, of which only PandaSet
  may be redistributed.
- The gates are frozen in `tests/*.gates.json` at the first green run and are never loosened.

## Limits

- Batch 1 on the chip; one model per process; one frame (one sweep) per call; one variant per model.
- The device input has a fixed size per profile (`N_cap` 160,000 / 120,000 points, `M_cap` 60,000 cells): a frame
  outside Autoware's profile ranges is skipped exactly as the node skips it.
- Label names 22 / 23: Autoware's `class_names` read `debris` / `stroller`, but the weights were trained with
  stroller = 22 and debris = 23 (AWML config). The port keeps Autoware's names for parity.
- Numerics: the device's per-point probabilities differ slightly from the fp32 reference, so points near a class
  boundary can take the other class (98.9-99.6 % of the points agree on the public frames, see the model card).
- Not ported: Autoware's `PointCloud2` layouts and the filtered cloud's `output_format` conversion; the API takes
  arrays or files and returns arrays.