OPT_REPORT.md

frnet-p150 optimization report (p150, ETH dispatch, 12×10 grid)

Status: baseline port; optimization pending. This is the first public release: the functional port, measured once (OPT_BASELINE.md, baseline commit 0dd52b2, 2026-10-09) and not optimized yet. No optimization round has run, so there is no step, no rejected attempt and no hang to report, and every number below is the baseline. The device graph runs as one metal trace per serve profile: 1,104 programs per frame for ot128 (Autoware's default sensor model) and 759 for qt128, 211.64 / 134.87 ms per back-to-back replay on a PandaSet frame (4.72 / 7.41 frames/s of device throughput). A synchronous model() call takes 292.6 ms (ot128) / 221.1 ms (qt128) on the shipped synthetic sample and 355.7 / 244.3 ms on the PandaSet frame; the device is 60 % (ot128) / 55 % (qt128) of it. The ranked backlog at the end says where the time goes. The largest item is the precision island of the point-wise layers (63 % of the ot128 replay, 75 % of qt128), whose cost is DRAM traffic of fp32 partial sums, not compute; then the image-branch convs and glue (46 ms) and the custom K1 segment-max kernel (16.9 ms). The binding accuracy constraint of every step is the nuScenes 0916 key-frame 40 frame on ot128: 98.907 % argmax agreement against the 98.5 % gate, a 0.41-point margin.

All numbers: code/scripts/bench.py (median of 100 warm iterations, batch 1, the shipped sample code/tt_frnet/samples/synthetic_<variant>.npz and the staged PandaSet 019 frame 40), ETH dispatch, 1 CQ, 12×10 grid, the serve.env precision knobs (FRNET_POINT_TERMS=2: two-term bf16 operands with fp32 accumulation in the point-wise layers; FRNET_MAP_DTYPE=float32: fp32 image-branch maps with bf16 conv operands; FRNET_CONV_TERMS=1), HiFi4 with fp32 accumulation everywhere, AICLK 1350 MHz, unless marked otherwise. Device profile: code/scripts/profile_ops.py under the device profiler (one traced replay between signposts). Accuracy gates: OPT_BASELINE.md "How to run".

Summary

	baseline 0dd52b2 (2026-10-09)	final (= baseline: no round yet)
device trace, one blocking forward (ot128 / qt128; shipped sample)	210.02 / 133.53 ms	same
back-to-back traces (ot128 / qt128; PandaSet 019 f40)	211.64 / 134.87 ms (4.72 / 7.41 frames/s)	same
e2e model() p50 (p99), shipped sample: ot128 / qt128	292.6 (299.4) / 221.1 (233.5) ms	same
e2e model() p50 (p99), PandaSet 019 f40 (107,146 points): ot128 / qt128	355.7 (367.0) / 244.3 (258.7) ms	same
host preprocess / H2D / D2H / un-permute / postprocess, ot128, PandaSet	107.2 / 5.41 / 7.43 / 12.8 / 14.8 ms	same
device programs per frame (distinct programs in the trace): ot128 / qt128	1,104 (227) / 759 (174)	same
kernel sum / span / op-to-op gaps, ot128	212.64 / 213.27 / 0.63 ms	same
kernel sum / gaps, qt128 (span unusable: a profiler artefact, see OPT_BASELINE.md)	134.81 / 0.44 ms	same
accuracy gates (vs the fp32 CPU reference / ONNX Runtime)	every frozen gate passes: module PCC min 0.999729 / 0.999649; public frames min pred_probs PCC 0.99969 / 0.99980, min argmax agreement 98.907 % (nuScenes 0916 kf40, gate 98.5 %) / 99.158 %	same
from_pretrained load, warm JIT cache	6.5 / 6.9 s (capture 2.6 / 2.0 s)	same

Host-side rows move with the load of the shared host (load average 4.1–6.6 during the baseline job); the device rows repeat to 0.01 ms. The release re-check (VERIFICATION_2026-10-09.md) reproduced the device rows on the release commit.

Steps (chronological; each row is one commit)

step	commit	what	trace / b2b ms (ot128; qt128)	e2e ms (shipped sample: ot128; qt128)	accuracy	revert switch
baseline	0dd52b2	first correct port: one trace per profile, ETH 1CQ 12×10; two-term bf16 point layers, fp32 image-branch maps, K1 segment_reduce (bit-exact); host pre-processing, argmax and filter	210.02 / 209.95; 133.53 / 133.43	292.6; 221.1	all frozen gates (above)	–

The release commit after the baseline changed no model numerics and no device code: the shipped sample became a synthetic scan per serve profile (generated by code/scripts/make_sample.py, byte-identical to the baseline's synthetic bench inputs), the PandaSet sample moved out of the shipped code/ (staged until PandaSet samples are approved), test hardening (test_public_frames fails on a missing golden and pins the frame counts; a new test_shipped_sample gate), docs and demo media. The device suite was re-run on it with identical values for every existing gate (VERIFICATION_2026-10-09.md).

Round 1

Not started. The optimization phase follows the publication of every model of the collection (research/PLAN.md §4.1, §5): each step is one commit with one FRNET_* A/B knob, keeps the frozen gates, and the precision policy (PLAN.md §9.3) may be relaxed per module only with gate evidence. Every numerics change is judged on the binding frame first: nuScenes 0916 kf40, ot128, 98.907 % against the 98.5 % gate. A change of the fp32 accumulation order is not bit-identical to today, so it re-runs the full device suite.

Findings (measured)

The baseline profile (OPT_BASELINE.md "Device profile") is the starting point:

• The trace is kernel-bound, not dispatch-bound: op-to-op gaps total 0.63 ms of a 213.27 ms span (ot128, 0.3 %). Cutting programs pays only through the kernel time of the programs removed; a megakernel's dispatch saving is at most 0.6 ms.
• The point-wise layers kept at higher precision are 63 % of ot128 (133.7 ms) and 75 % of qt128 (101.1 ms). Each linear splits its fp32 input into two bf16 terms and runs 2–3 bf16 matmuls whose fp32 partials are summed in fp32. The cost is DRAM traffic: 61.2 ms of matmuls at 1.4–7.9 % of the HiFi4 FLOPs, 52.6 ms of fp32 adds of the partials, 19.5 ms of typecasts for the splits. decoder.point_fuse1 alone is 39.9 ms.
• K1 segment_reduce: 16.9 ms (8 % of ot128, 11 % of qt128): 2.39 ms per 128-channel site almost independent of the row count, so per-segment output overhead dominates.
• Image-branch convs and glue: 46 ms on ot128, of which 12.6 ms conv kernels; the rest is ttnn's DRAM slicing of the 128 × 1024 convs (9.1 ms), halo / move / resharding (4.8 ms), 46 fp32 HardSwish (6.3 ms), bf16 casts before every conv (5.2 ms), fp32 residual and K-split adds (6.8 ms), resizes (9.0 ms).
• Every point-wise op runs on the profile's N_cap rows (160,000 / 120,000) and K1 writes all 60,000 segments, whatever the frame: the nuScenes frames (N = 20–28 k) cost the same as PandaSet (N = 146 k).
• L1 holds nothing between ops; DRAM use peaks at 218.6 MB per bank at the first partial-sum add of decoder.point_fuse1.
• Dispatch / CQ matrix (back-to-back): ot128 ETH-1CQ 209.95, ETH-2CQ 211.32, WORKER 211.56 ms; qt128 133.44, 134.25, 153.71 ms. ETH and 1 CQ stay pinned (D14): on qt128's narrow grid WORKER's 11×10 partitioning costs 20.3 ms.
• The precision policy is how the port reaches the gate. Measured in the port (PORT_LOG M1–M3): an fp32 × fp32 point island (the device's TF32-like matmul) replays in 107 ms but gives 97.50 % on the binding frame; bf16 image maps 98.28 %; CONV_TERMS=2 buys +0.09 pt for +37 ms.

Megakernel / fusion work

None yet. The candidates by measured block time are listed in the backlog (item 13).

Known hangs (all rounds)

when (UTC)	command	cause	status
–	–	none: no hang, device timeout, reset or FAULT marker in any device job of the port, its verification round, the baseline or the release runs (one build-time TT_THROW of the shared C18 resize at fp32 1024 × 128 rows, an L1 overflow reported at program build, is a known issue in PORT_LOG.md, not a hang)	–

Rejected / not kept

None yet in the optimization phase. Configurations measured during the port that are not the default, for accuracy (PORT_LOG.md M1–M3; OPT_BASELINE.md "The precision cost of the policy"; ot128, binding frame nuScenes 0916 kf40):

configuration	replay	binding frame	why not
fp32 × fp32 point island (POINT_TERMS=1), bf16 maps	107 ms	97.50 %	fails the 98.5 % gate
two-term point island, bf16 maps (MAP_DTYPE=bfloat16)	201 ms	98.28 %	fails
two-term point island, bf16 maps, CONV_TERMS=2	244 ms	98.35 %	fails, slower
two-term point island, fp32 maps fed to the convs directly (MAP_DTYPE=float32_direct)	217 ms	98.13 %	fails
two-term point island, fp32 maps, CONV_TERMS=2	249 ms	99.00 %	passes, +37 ms for +0.09 pt of margin

Profile at the end (trace replay)

The baseline profile (OPT_BASELINE.md): ot128 1,104 programs, kernel sum 212.64 ms, span 213.27 ms; qt128 759 programs, 134.81 ms. Top sinks by op type (ot128): Matmul 64.1 ms (30 %), BinaryNg 59.4 ms (28 %), Typecast 26.8 ms (13 %), K1 16.9 ms (8 %), Conv2d 12.6 ms (6 %), DRAM slicing 9.1 ms, HardSwish 6.3 ms, resize chains 7.3 ms.

Remaining backlog (gains on the ot128 trace of 211.6 ms, PandaSet, unless marked e2e; estimates, not measurements)

Each gain is the measured kernel or host time of the work an item removes, with a range for what replaces it. Items overlap, so the gains do not add up. qt128 follows the same ranking. From OPT_BASELINE.md "Ranked optimization opportunities":

1. Two-term point linears as one K-concatenated matmul per layer (trace −60..−80 ms). One matmul over [x_hi | x_hi | x_lo] × [W_hi; W_lo; W_hi] with fp32 dest and packer L1 accumulation replaces 2–3 matmuls per part and their fp32 partial adds. Same exact bf16 products; only the fp32 summation order changes (re-run the gates).
2. MK-PointMLP, a row-block-streamed point MLP megakernel (trace −100..−115 ms beyond today, overlapping 1; D19). The point layers are row-local except the gathers and K1; their bf16 weight terms fit L1 (at most 655 KB). Chain the encoder, the point fuse and the head per row block in L1, with the two-term split on the fly.
3. A fused two-term split (trace −10..−14 ms; on the way to 1–2). One pass reads fp32 and writes the hi / lo bf16 terms (exact: the same RNE casts) instead of 4 programs per input.
4. N_cap / M_cap buckets (trace −100..−160 ms on frames with N ≤ 32–64 k, e.g. every nuScenes frame). Two or three trace variants per profile sharing weights; the host picks the smallest that covers the frame. PandaSet-like frames (N = 146 k) gain nothing.
5. K1 throughput (trace −10..−14 ms; owned shared kernel). Batch output rows per worker, accumulator in the RISC-V local memory, then a compute-engine reduction; bit-exactness stays.
6. Image-branch DRAM slicing and glue (trace −8..−14 ms; MK-ImageStage). Keep the stride-1 stage chain height-sharded in L1 (a 128 × 1024 × 128 map is 33.5 MB in bf16), fold the bf16 cast and the HardSwish into one pass.
7. Decoder resizes and fuse1 (trace −5..−8 ms). Resize matmuls in TILE layout end to end (also lifts the C18 fp32 limitation), K-split accumulation inside the conv.
8. pixel2point gathers (trace −2..−4 ms). Feed the bf16 gather output straight into item 1's operand (exact).
9. Host pre-processing (e2e −60..−85 ms). 107 ms on PandaSet: a C++ / numba implementation of the node's CUDA kernels (projection + interpolation grid in one pass, counting-sort plan), bit-identical to today's host code.
10. Host after the device (e2e −15..−25 ms). Row-major readback of only N rows; argmax over the permuted rows, then scatter labels and scores instead of 27-column float rows (exact: the same first-max rule).
11. H2D features layout (e2e −3.5 ms). 160,000 pages of 32 bytes at 1.24 GB/s: upload as 128-byte rows.
12. Throughput by pipelining (frames/s; latency unchanged). Overlap the host stages of neighbouring frames with the device work; 2 CQ staging once items 1–2 shrink the replay.
13. Megakernel candidates (D19, PLAN §5.2 / §5.3) by measured block time: MK-PointMLP (133.7 / 101.1 ms; first), MK-SegGather (K1 + gathers, 24 ms), MK-ImageStage (46 ms). A whole-model kernel is not practical: the fp32 point tensors at N_cap (82–164 MB each) do not fit L1.
14. Precision margin (accuracy, an optimization-phase precision item). The 0.41-point margin on nuScenes 0916 kf40 (ot128) is the narrowest of the release. Items 1–2 change the accumulation order, so re-check it after each step. Measured option: CONV_TERMS=2 (+0.09 pt, +37 ms). Unmeasured: fp32 resize outputs (after item 7), fp32 gathers of the 128-channel maps. No relaxation (bf16 maps, a TF32 point island, HiFi2, bfp8) is a candidate without new device evidence on that frame.