PYTHON.md

Python API: FRNet (Autoware lidar_frnet) on Blackhole

Use this API from Python code (a pipeline, a notebook, a ROS 2 node wrapper). You do not need the HTTP server: the API and the server share the decoders, the device graph and the post-processing, so the outputs and the speed are the same.

Install

Install the package on top of an environment that already has ttnn (a tt-metal python_env at 44d66500520 with patches/tt-metal-eth-dispatch.patch, or the tt-model container). From the root of the model repository (the directory that holds pyproject.toml, README.md and code/):

pip install -e .                        # the Python API
pip install -e ".[server,test]"         # + the HTTP server and the tests

The pip project is the repository's top-level pyproject.toml; it installs the package from code/tt_frnet (there is no pyproject.toml inside code/, because the container build copies code/ over the tt-metal tree).

The package carries tt_frnet.ttaw, the shared code of the Autoware ports to Blackhole (device open, trace runner, decoders, model base class, HTTP app, the K1 segment_reduce kernel), vendored at the version recorded in code/tt_frnet/ttaw/VENDORED.json (0.20.0).

You want to run	Extras
the Python API	none
the HTTP server (tt_frnet.server.app, see SERVING.md)	server
host tests (no device; device tests are skipped): TT_VISIBLE_DEVICES=none python -m pytest -q code/tt_frnet/tests	server,test
device tests: python -m pytest -q -s code/tt_frnet/tests/test_pcc_device.py code/tt_frnet/tests/test_e2e_device.py	test

Quickstart

Run from the repository root (the sample paths are relative to it):

from tt_frnet import FRNet

with FRNet.from_pretrained(device_id=0) as model:                  # variant="qt128" for the Hesai QT128 profile
    out = model("code/tt_frnet/samples/synthetic_ot128.npz",       # one sweep in the sensor frame
                calibration={"preset": "synthetic_ot128"})         # its sensor -> base_link transform (ego crop box)

print(out.class_counts())                                          # points per class (raw + interpolated)
labels, scores, xyz = out.label_ids, out.scores, out.points[:, :3]

examples/quickstart.py runs this snippet and writes quickstart.json (the /predict body) and quickstart_bev.png (a bird's-eye view of the p150 classes next to the stored CPU reference of the shipped sample).

FRNet.from_pretrained(...)

FRNet.from_pretrained(
    model_id=None,          # HF repo or a local directory with the weights files; default AutowareFoundation/lidar_frnet
    *,
    revision=None,          # default for the default repo: the validated commit a53a1c11b0b (tag v2.0)
    variant=None,           # load-time variant: "ot128" (Hesai OT128, Autoware's default) | "qt128"; default $FRNET_VARIANT or "ot128"
    device_id=None,         # chip to open; default $TT_DEVICE_ID or 0
    device=None,            # an already-opened ttnn device (tt_frnet.device.open_device); close() does not close it
    dispatch=None,          # "eth" (p150 target, 12x10 grid) | "worker" (A/B only, 11x10) | "auto"; default $FRNET_DISPATCH or "eth"
    num_command_queues=None,  # default $FRNET_NUM_CQS or 1
    weights_dir=None,       # explicit local weights directory; no Hub access
    warmup_variants="default",  # trace variants to capture now; see "Warm-up"
    verbose=False,
) -> FRNet

What it does: resolves the weights first, so a Hub problem never claims the chip (weights_dir > $FRNET_WEIGHTS_DIR > a local model_id directory > the HF snapshot at the pinned revision, restricted to frnet_*.onnx, ml_package_frnet_*.param.yaml, deploy_metadata.yaml, with an offline fallback to the cache), checks the sha256 of the variant's ONNX against the pinned one, reads the variant's ml_package_frnet_<variant>.param.yaml (FOV, frustum and interpolation grids, the 27 class names, the palette, the point / cell profiles), opens the chip (ETH dispatch, 12×10, 1 CQ, l1_small_size 64 KiB, trace_region_size 256 MiB: DEVICE_DEFAULTS in tt_frnet/device.py, overridable with FRNET_*), builds the graph, then compiles and captures the metal trace. If ETH dispatch cannot open (tt-metal without the patch), it warns and falls back to WORKER dispatch.

variant picks one of the two networks of the weights repo. Each is tied to its sensor (Autoware's sensor_model):

variant	sensor	FOV up / down	frustum grid (H × W)	interpolation grid	points (min / max)	unique cells (min / max)
ot128 (default)	Hesai OT128	+15° / −25°	128 × 1024	128 × 4096	5,000 / 160,000	3,000 / 60,000
qt128	Hesai QT128	+52.6° / −52.6°	128 × 256	128 × 1024	5,000 / 120,000	3,000 / 60,000

Precision knobs (read once at load from the environment, pinned in the package's serve.env; the defaults are the validated configuration and every number of the model card uses them): FRNET_POINT_TERMS=2 (each point-wise linear as two-term bf16 operands, about 16 mantissa bits, fp32 accumulation), FRNET_MAP_DTYPE=float32 (fp32 image-branch maps, bf16 conv operands), FRNET_CONV_TERMS=1. Other values are A/B switches for the optimization phase; they fail the frozen accuracy gates (OPT_BASELINE.md, "The precision cost of the policy").

Warm-up

from_pretrained returns a warm model: it builds the graph of the variant, runs it once eagerly on an empty frame (this first run compiles the kernels and uploads the weights) and captures it as ONE metal trace, so no call compiles anything. There are no shape buckets: every frame is padded to the profile's fixed device shapes (N_cap points = 160,000 / 120,000, M_cap cells = 60,000), so one trace serves every input of the profile.

• Measured (2026-10-09, the shipped sample): with an empty JIT cache the load took 156.6 s for ot128 and 151.9 s for qt128 (firmware and every kernel compiled); with that cache warm, in a new process, 7.5 s and 6.6 s. The trace is captured during the load, so the first call costs about as much as the next ones (320.0 / 304.7 ms for ot128, 241.1 / 214.9 ms for qt128, first / second call, on a shared host).
• warmup_variants="none" (server: FRNET_WARMUP=none) skips the capture at load; call model.warmup() before the first frame (it is idempotent).
• The JIT cache is tt-metal's (~/.cache/tt-metal-cache, or TT_METAL_CACHE); inside the container it is ~/.cache/tt-model/frnet-p150/cache of the host.

Call: model(points, calibration=..., **kwargs)

Argument	Type	Description
points (first positional)	path, bytes + fmt=, (N, C) array / tensor, structured array, PointCloud, JSON envelope	ONE LiDAR sweep in the sensor frame (x forward, y left, z up, metres), fields x, y, z, intensity with the raw intensity the node casts (0-255 for uint8 drivers); see "Input types"
calibration	{"T_base_link_from_sensor": <transform>}, {"preset": name} (a file of code/tt_frnet/calib/), optional "ego_crop_box"	the static sensor -> base_link transform the node looks up once for its ego crop box. ego_crop_box: [min_x, min_y, min_z, max_x, max_y, max_z] in base_link (default Autoware's (-1.53, -1.15, -0.1) .. (6.1, 1.15, 3.1)), or None (or all zeros) to disable the box, the node's own switch; a disabled box needs no transform
class_probability_threshold	float in [0, 1], default 0.05	the filtered cloud keeps a point unless its probability of a filter class is >= this value (Autoware's filter.class_probability_threshold)
filter_classes	comma-separated class names, "" = none, default "drivable_surface"	Autoware's filter.classes (at most 16); an unknown name is refused

Both knobs only change filter_keep; the class decision (label_ids, scores) never depends on them. They are host-only (no device shape changes), the defaults reproduce Autoware's frnet.param.yaml, and a bad value raises tt_frnet.io.InputError (HTTP 400) before the device runs. model.info["runtime_params"] (and the server's /info) lists them with their defaults.

Input types

• Point clouds: a path (.bin raw float32 rows -- pass fields= if not x, y, z, intensity; .npy; .npz; .pcd ascii / binary / binary_compressed), raw bytes with fmt=, an (N, C) numpy array or torch tensor, a structured array, the JSON envelope of /predict, or tt_frnet.io.PointCloud. The frame is the sensor frame (Autoware's ~/input/pointcloud before any transform), one sweep, not motion-compensated.
• Rows with a non-finite x / y / z are dropped by the decoder; input_index refers to the decoded rows.
• Calibration transforms: a 4×4 (or 3×4) row-major matrix, {"translation": [x, y, z], "rotation_wxyz": [w, x, y, z]}, or Autoware's {x, y, z, roll, pitch, yaw}. With the crop box on and no transform the call is refused (the node would skip every frame because its TF lookup fails); the error names both fixes.
• sweeps, images, stream and inputs are refused: FRNet takes one sweep and its calibration only.
• A frame outside the variant's profile (input points, points after interpolation, or unique cells out of the table above) is not an error: as the node, which logs and publishes nothing, the call returns an empty output whose meta["skipped"] holds the node's message, and nothing runs on the device.

Output

model(...) returns a tt_frnet.FRNetSegmentation (a ttaw.outputs.Segmentation3D); out.to_dict() is exactly the POST /predict body (SERVING.md section 3.2). The point order is the node's: the input points that survive the ego crop box, in input order, then the points Autoware's interpolation appends (D4: published as Autoware does).

field	type	meaning
label_ids	uint8 [N]	Autoware's class_id (first maximum of the 27 probabilities, as fill_cloud_kernel); names in class_names
scores	float32 [N]	Autoware's probability: the softmax probability of that class
points	float32 [N, 4]	x, y, z, intensity of every output point in the sensor frame (the segmentation cloud's coordinates; the interpolated points carry interpolated values)
filter_keep	bool [N]	membership of Autoware's ~/output/pointcloud/filtered cloud (False = removed by the class filter)
num_points_raw	int	the first num_points_raw rows are input points, the rest are interpolated
input_index	int32 [num_points_raw]	the decoded input row of every raw output point
class_names	27 str	drivable_surface, other_flat_surface, sidewalk, manmade, vegetation, car, bus, emergency_vehicle, train, truck, tractor_unit, semi_trailer, construction_vehicle, forklift, kart, motorcycle, bicycle, pedestrian, personal_mobility, animal, pushable_pullable, traffic_cone, debris, stroller, other_stuff, noise+ghost_point, unknown (Autoware's names; ids 22 / 23 were trained as stroller / debris, see Limits)
meta	dict	variant; num_points_input; ego_crop (the box and transform used); the network-input counts (num_points_raw, num_points_interpolated, num_points, num_unique_coors, cropped_by_ego_box, crop_applied, max_points_per_cell); plan (n, m, n_cap, m_cap); filter (classes, threshold, points kept); skipped (the node's message) for an out-of-profile frame
timing_ms	dict	preprocess, device (staging + H2D + trace + D2H + un-permute), postprocess, total of this call

out.class_counts() gives the points per class (raw + interpolated); out.to_dicts() one row per class (label, label_id, points); len(out) the number of output points. The node's visualization cloud is model.config.palette indexed by label_ids (tt_frnet.host.postprocess.visualization_rgb).

Lifetime and information

• model.close() releases the trace and the device tensors and closes the chip if the model opened it; idempotent. with calls it for you; an unclosed model is closed when Python exits.
• model.info: weights (repo, tag, revision, path), device (dispatch, grid, cores, num_command_queues, fallback), variant, warm variants, warm-up times, runtime parameter defaults, the variant config (frnet), the trace and graph descriptions.
• Calls from several threads are safe: the device calls are serialised by a lock. One model per process per chip.
• tt_frnet.viz.render_bev(points, label_ids, palette=...) and render_compare(...) draw bird's-eye views (PIL, no OpenCV).
• model.device_probs(plan) returns the device's pred_probs (N, 27) of a prepared frame (tests and tools; call it under model._lock).

Speed

Warm calls, batch 1, ETH dispatch, 1 CQ, 12x10, the serve.env precision knobs (code/scripts/bench.py, 100 iterations; the numbers of OPT_BASELINE.md, 2026-10-09, on a shared host whose load moves the host stages):

stage, p50	ot128, shipped sample	ot128, PandaSet 019 f40	qt128, shipped sample	qt128, PandaSet 019 f40
model(path) end to end	292.6 ms	355.7 ms	221.1 ms	244.3 ms
host pre-processing (decode, crop, projection, fp16 interpolation, unique cells, device plan)	55.3 ms	107.2 ms	54.2 ms	83.5 ms
H2D (9 inputs, 8.6 / 6.2 MB)	5.42 ms	5.41 ms	4.04 ms	4.00 ms
device trace, one forward	210.02 ms	211.72 ms	133.53 ms	134.91 ms
D2H (fp32 pred_probs)	7.65 ms	7.43 ms	5.80 ms	5.55 ms
un-permute + host post-processing (argmax, filter)	7.1 + 5.0 ms	12.8 + 14.8 ms	6.1 + 6.8 ms	5.9 + 10.5 ms

• Back-to-back replays take 211.64 ms (ot128) and 134.87 ms (qt128) per PandaSet frame: 4.72 and 7.41 frames/s of device throughput. A synchronous call gives 2.81 / 4.09 frames/s on that frame.
• The device time barely depends on the frame: every point-wise op runs on the profile's N_cap rows. The host stages grow with the cloud.
• The shipped samples are synthetic scans: ot128 53,411 points (N = M = 52,613 after the crop, no interpolated point), qt128 45,783 points (N = 69,553 after interpolation, M = 18,926). PandaSet 019 frame 40 is a 107,146-point Pandar64 sweep (N = 145,961 / M = 40,691 on ot128); it is not shipped.
• OPT_REPORT.md ranks what the optimization phase will change (the two-term point layers, K1, the image-branch glue, shape buckets, the host pre-processing).

Tests

From the repository root, with code/ on PYTHONPATH (or the package installed):

• Host (no device, no ttnn call): TT_VISIBLE_DEVICES=none python -m pytest -q code/tt_frnet/tests. Inside the tt-models workspace the suite runs on its fake ttnn (-p fake_ttnn_plugin), which also enables test_tt_host.py; tests that need workspace-only oracles or data skip elsewhere, and FAIL inside the workspace when that data is missing. test_reference_cpu.py needs onnxruntime.
• Device (one p150): python -m pytest -q -s code/tt_frnet/tests/test_pcc_device.py code/tt_frnet/tests/test_e2e_device.py (per-module PCC against the fp32 CPU reference on replay outputs, the K1 segment-max sites included; end-to-end agreement with ONNX Runtime on 17 OT128 / 5 QT128 frames, the shipped samples against their stored references, replay == eager, API == /predict). The public-frame gates run only inside the tt-models workspace, where their data lives: PandaSet, nuScenes and Autoware's sample rosbag, of which only PandaSet may be redistributed.
• The gates are frozen in tests/*.gates.json at the first green run and are never loosened.

Limits

• Batch 1 on the chip; one model per process; one frame (one sweep) per call; one variant per model.
• The device input has a fixed size per profile (N_cap 160,000 / 120,000 points, M_cap 60,000 cells): a frame outside Autoware's profile ranges is skipped exactly as the node skips it.
• Label names 22 / 23: Autoware's class_names read debris / stroller, but the weights were trained with stroller = 22 and debris = 23 (AWML config). The port keeps Autoware's names for parity.
• Numerics: the device's per-point probabilities differ slightly from the fp32 reference, so points near a class boundary can take the other class (98.9-99.6 % of the points agree on the public frames, see the model card).
• Not ported: Autoware's PointCloud2 layouts and the filtered cloud's output_format conversion; the API takes arrays or files and returns arrays.