OPT_REPORT.md

# frnet-p150 optimization report (p150, ETH dispatch, 12×10 grid)

**Status: baseline port; optimization pending.** This is the first public release: the functional port, measured
once (`OPT_BASELINE.md`, baseline commit `0dd52b2`, 2026-10-09) and not optimized yet. No optimization round has run,
so there is no step, no rejected attempt and no hang to report, and every number below is the baseline. The device
graph runs as one metal trace per serve profile: 1,104 programs per frame for `ot128` (Autoware's default sensor
model) and 759 for `qt128`, 211.64 / 134.87 ms per back-to-back replay on a PandaSet frame (4.72 / 7.41 frames/s of
device throughput). A synchronous `model()` call takes 292.6 ms (ot128) / 221.1 ms (qt128) on the shipped synthetic
sample and 355.7 / 244.3 ms on the PandaSet frame; the device is 60 % (ot128) / 55 % (qt128) of it. The ranked
backlog at the end says where the time goes. The largest item is the precision island of the point-wise layers
(63 % of the ot128 replay, 75 % of qt128), whose cost is DRAM traffic of fp32 partial sums, not compute; then the
image-branch convs and glue (46 ms) and the custom K1 segment-max kernel (16.9 ms). The binding accuracy constraint
of every step is the nuScenes 0916 key-frame 40 frame on ot128: 98.907 % argmax agreement against the 98.5 % gate,
a 0.41-point margin.

All numbers: `code/scripts/bench.py` (median of 100 warm iterations, batch 1, the shipped sample
`code/tt_frnet/samples/synthetic_<variant>.npz` and the staged PandaSet 019 frame 40), ETH dispatch, 1 CQ, 12×10 grid,
the serve.env precision knobs (`FRNET_POINT_TERMS=2`: two-term bf16 operands with fp32 accumulation in the point-wise
layers; `FRNET_MAP_DTYPE=float32`: fp32 image-branch maps with bf16 conv operands; `FRNET_CONV_TERMS=1`), HiFi4 with
fp32 accumulation everywhere, AICLK 1350 MHz, unless marked otherwise. Device profile: `code/scripts/profile_ops.py`
under the device profiler (one traced replay between signposts). Accuracy gates: `OPT_BASELINE.md` "How to run".

## Summary

| | baseline `0dd52b2` (2026-10-09) | **final (= baseline: no round yet)** |
|---|---|---|
| device trace, one blocking forward (ot128 / qt128; shipped sample) | 210.02 / 133.53 ms | same |
| back-to-back traces (ot128 / qt128; PandaSet 019 f40) | 211.64 / 134.87 ms (4.72 / 7.41 frames/s) | same |
| e2e `model()` p50 (p99), shipped sample: ot128 / qt128 | 292.6 (299.4) / 221.1 (233.5) ms | same |
| e2e `model()` p50 (p99), PandaSet 019 f40 (107,146 points): ot128 / qt128 | 355.7 (367.0) / 244.3 (258.7) ms | same |
| host preprocess / H2D / D2H / un-permute / postprocess, ot128, PandaSet | 107.2 / 5.41 / 7.43 / 12.8 / 14.8 ms | same |
| device programs per frame (distinct programs in the trace): ot128 / qt128 | 1,104 (227) / 759 (174) | same |
| kernel sum / span / op-to-op gaps, ot128 | 212.64 / 213.27 / 0.63 ms | same |
| kernel sum / gaps, qt128 (span unusable: a profiler artefact, see `OPT_BASELINE.md`) | 134.81 / 0.44 ms | same |
| accuracy gates (vs the fp32 CPU reference / ONNX Runtime) | every frozen gate passes: module PCC min 0.999729 / 0.999649; public frames min `pred_probs` PCC 0.99969 / 0.99980, min argmax agreement **98.907 %** (nuScenes 0916 kf40, gate 98.5 %) / 99.158 % | same |
| `from_pretrained` load, warm JIT cache | 6.5 / 6.9 s (capture 2.6 / 2.0 s) | same |

Host-side rows move with the load of the shared host (load average 4.1–6.6 during the baseline job); the device rows
repeat to 0.01 ms. The release re-check (`VERIFICATION_2026-10-09.md`) reproduced the device rows on the release
commit.

## Steps (chronological; each row is one commit)

| step | commit | what | trace / b2b ms (ot128; qt128) | e2e ms (shipped sample: ot128; qt128) | accuracy | revert switch |
|---|---|---|---|---|---|---|
| baseline | `0dd52b2` | first correct port: one trace per profile, ETH 1CQ 12×10; two-term bf16 point layers, fp32 image-branch maps, K1 `segment_reduce` (bit-exact); host pre-processing, argmax and filter | 210.02 / 209.95; 133.53 / 133.43 | 292.6; 221.1 | all frozen gates (above) | – |

The release commit after the baseline changed no model numerics and no device code: the shipped sample became a
synthetic scan per serve profile (generated by `code/scripts/make_sample.py`, byte-identical to the baseline's
synthetic bench inputs), the PandaSet sample moved out of the shipped `code/` (staged until PandaSet samples are
approved), test hardening (`test_public_frames` fails on a missing golden and pins the frame counts; a new
`test_shipped_sample` gate), docs and demo media. The device suite was re-run on it with identical values for every
existing gate (`VERIFICATION_2026-10-09.md`).

## Round 1

Not started. The optimization phase follows the publication of every model of the collection (`research/PLAN.md`
§4.1, §5): each step is one commit with one `FRNET_*` A/B knob, keeps the frozen gates, and the precision policy
(PLAN.md §9.3) may be relaxed per module only with gate evidence. **Every numerics change is judged on the binding
frame first: nuScenes 0916 kf40, ot128, 98.907 % against the 98.5 % gate.** A change of the fp32 accumulation order
is not bit-identical to today, so it re-runs the full device suite.

### Findings (measured)

The baseline profile (`OPT_BASELINE.md` "Device profile") is the starting point:

- The trace is **kernel-bound, not dispatch-bound**: op-to-op gaps total 0.63 ms of a 213.27 ms span (ot128, 0.3 %).
  Cutting programs pays only through the kernel time of the programs removed; a megakernel's dispatch saving is at
  most 0.6 ms.
- **The point-wise layers kept at higher precision are 63 % of ot128 (133.7 ms) and 75 % of qt128 (101.1 ms).** Each
  linear splits its fp32 input into two bf16 terms and runs 2–3 bf16 matmuls whose fp32 partials are summed in fp32.
  The cost is DRAM traffic: 61.2 ms of matmuls at 1.4–7.9 % of the HiFi4 FLOPs, 52.6 ms of fp32 adds of the
  partials, 19.5 ms of typecasts for the splits. `decoder.point_fuse1` alone is 39.9 ms.
- **K1 `segment_reduce`: 16.9 ms** (8 % of ot128, 11 % of qt128): 2.39 ms per 128-channel site almost independent of
  the row count, so per-segment output overhead dominates.
- **Image-branch convs and glue: 46 ms on ot128**, of which 12.6 ms conv kernels; the rest is ttnn's DRAM slicing of
  the 128 × 1024 convs (9.1 ms), halo / move / resharding (4.8 ms), 46 fp32 HardSwish (6.3 ms), bf16 casts before
  every conv (5.2 ms), fp32 residual and K-split adds (6.8 ms), resizes (9.0 ms).
- Every point-wise op runs on the profile's `N_cap` rows (160,000 / 120,000) and K1 writes all 60,000 segments,
  whatever the frame: the nuScenes frames (N = 20–28 k) cost the same as PandaSet (N = 146 k).
- L1 holds nothing between ops; DRAM use peaks at 218.6 MB per bank at the first partial-sum add of
  `decoder.point_fuse1`.
- Dispatch / CQ matrix (back-to-back): ot128 ETH-1CQ 209.95, ETH-2CQ 211.32, WORKER 211.56 ms; qt128 133.44, 134.25,
  153.71 ms. ETH and 1 CQ stay pinned (D14): on qt128's narrow grid WORKER's 11×10 partitioning costs 20.3 ms.
- The precision policy is how the port reaches the gate. Measured in the port (PORT_LOG M1–M3): an fp32 × fp32 point
  island (the device's TF32-like matmul) replays in 107 ms but gives 97.50 % on the binding frame; bf16 image maps
  98.28 %; `CONV_TERMS=2` buys +0.09 pt for +37 ms.

### Megakernel / fusion work

None yet. The candidates by measured block time are listed in the backlog (item 13).

## Known hangs (all rounds)

| when (UTC) | command | cause | status |
|---|---|---|---|
| – | – | none: no hang, device timeout, reset or FAULT marker in any device job of the port, its verification round, the baseline or the release runs (one build-time `TT_THROW` of the shared C18 resize at fp32 1024 × 128 rows, an L1 overflow reported at program build, is a known issue in `PORT_LOG.md`, not a hang) | – |

## Rejected / not kept

None yet in the optimization phase. Configurations measured during the port that are **not** the default, for
accuracy (`PORT_LOG.md` M1–M3; `OPT_BASELINE.md` "The precision cost of the policy"; ot128, binding frame nuScenes
0916 kf40):

| configuration | replay | binding frame | why not |
|---|---:|---:|---|
| fp32 × fp32 point island (`POINT_TERMS=1`), bf16 maps | 107 ms | 97.50 % | fails the 98.5 % gate |
| two-term point island, bf16 maps (`MAP_DTYPE=bfloat16`) | 201 ms | 98.28 % | fails |
| two-term point island, bf16 maps, `CONV_TERMS=2` | 244 ms | 98.35 % | fails, slower |
| two-term point island, fp32 maps fed to the convs directly (`MAP_DTYPE=float32_direct`) | 217 ms | 98.13 % | fails |
| two-term point island, fp32 maps, `CONV_TERMS=2` | 249 ms | 99.00 % | passes, +37 ms for +0.09 pt of margin |

## Profile at the end (trace replay)

The baseline profile (`OPT_BASELINE.md`): ot128 1,104 programs, kernel sum 212.64 ms, span 213.27 ms; qt128 759
programs, 134.81 ms. Top sinks by op type (ot128): Matmul 64.1 ms (30 %), BinaryNg 59.4 ms (28 %), Typecast 26.8 ms
(13 %), K1 16.9 ms (8 %), Conv2d 12.6 ms (6 %), DRAM slicing 9.1 ms, HardSwish 6.3 ms, resize chains 7.3 ms.

## Remaining backlog (gains on the ot128 trace of 211.6 ms, PandaSet, unless marked e2e; estimates, not measurements)

Each gain is the measured kernel or host time of the work an item removes, with a range for what replaces it. Items
overlap, so the gains do not add up. qt128 follows the same ranking. From `OPT_BASELINE.md` "Ranked optimization
opportunities":

1. **Two-term point linears as one K-concatenated matmul per layer (trace −60..−80 ms).** One matmul over
   `[x_hi | x_hi | x_lo] × [W_hi; W_lo; W_hi]` with fp32 dest and packer L1 accumulation replaces 2–3 matmuls per part
   and their fp32 partial adds. Same exact bf16 products; only the fp32 summation order changes (re-run the gates).
2. **MK-PointMLP, a row-block-streamed point MLP megakernel (trace −100..−115 ms beyond today, overlapping 1; D19).**
   The point layers are row-local except the gathers and K1; their bf16 weight terms fit L1 (at most 655 KB). Chain
   the encoder, the point fuse and the head per row block in L1, with the two-term split on the fly.
3. **A fused two-term split (trace −10..−14 ms; on the way to 1–2).** One pass reads fp32 and writes the hi / lo bf16
   terms (exact: the same RNE casts) instead of 4 programs per input.
4. **`N_cap` / `M_cap` buckets (trace −100..−160 ms on frames with N ≤ 32–64 k, e.g. every nuScenes frame).** Two or
   three trace variants per profile sharing weights; the host picks the smallest that covers the frame. PandaSet-like
   frames (N = 146 k) gain nothing.
5. **K1 throughput (trace −10..−14 ms; owned shared kernel).** Batch output rows per worker, accumulator in the
   RISC-V local memory, then a compute-engine reduction; bit-exactness stays.
6. **Image-branch DRAM slicing and glue (trace −8..−14 ms; MK-ImageStage).** Keep the stride-1 stage chain
   height-sharded in L1 (a 128 × 1024 × 128 map is 33.5 MB in bf16), fold the bf16 cast and the HardSwish into one pass.
7. **Decoder resizes and `fuse1` (trace −5..−8 ms).** Resize matmuls in TILE layout end to end (also lifts the C18 fp32
   limitation), K-split accumulation inside the conv.
8. **pixel2point gathers (trace −2..−4 ms).** Feed the bf16 gather output straight into item 1's operand (exact).
9. **Host pre-processing (e2e −60..−85 ms).** 107 ms on PandaSet: a C++ / numba implementation of the node's CUDA
   kernels (projection + interpolation grid in one pass, counting-sort plan), bit-identical to today's host code.
10. **Host after the device (e2e −15..−25 ms).** Row-major readback of only N rows; argmax over the permuted rows, then
    scatter labels and scores instead of 27-column float rows (exact: the same first-max rule).
11. **H2D `features` layout (e2e −3.5 ms).** 160,000 pages of 32 bytes at 1.24 GB/s: upload as 128-byte rows.
12. **Throughput by pipelining (frames/s; latency unchanged).** Overlap the host stages of neighbouring frames with the
    device work; 2 CQ staging once items 1–2 shrink the replay.
13. **Megakernel candidates (D19, PLAN §5.2 / §5.3)** by measured block time: MK-PointMLP (133.7 / 101.1 ms; first),
    MK-SegGather (K1 + gathers, 24 ms), MK-ImageStage (46 ms). A whole-model kernel is not practical: the fp32 point
    tensors at `N_cap` (82–164 MB each) do not fit L1.
14. **Precision margin (accuracy, an optimization-phase precision item).** The 0.41-point margin on nuScenes 0916 kf40
    (ot128) is the narrowest of the release. Items 1–2 change the accumulation order, so re-check it after each step.
    Measured option: `CONV_TERMS=2` (+0.09 pt, +37 ms). Unmeasured: fp32 resize outputs (after item 7), fp32 gathers of
    the 128-channel maps. No relaxation (bf16 maps, a TF32 point island, HiFi2, bfp8) is a candidate without new device
    evidence on that frame.