OPT_BASELINE.md

frnet-p150 baseline on the p150 (before optimization)

Date 2026-10-09. The model code is the port's final commit 14fa968 (M2-M4; VERIFY_PORT round 1: PASS), unchanged. The baseline commits add only the measurement scripts (c246f92: code/scripts/bench.py, profile_ops.py, profile_summary.py, visualizer_report.py; this commit: visualizer_report.py --json-dir) and this file. tt-metal 44d66500520 + patches/tt-metal-eth-dispatch.patch. Weights AutowareFoundation/lidar_frnet @ a53a1c11b0b (tag v2.0), read from the sha256-pinned local copy through FRNET_WEIGHTS_DIR.

Both serve profiles are measured: ot128 (default; Hesai OT128, 128 × 1024 frustum grid, N_cap 160,000 points, M_cap 60,000 cells) and qt128 (Hesai QT128, 128 × 256 grid, N_cap 120,000, M_cap 60,000). The device graph has fixed shapes per profile (N_cap, M_cap). So the point-wise device work does not depend on the frame's point count, while the host rows and the data-dependent K1 segment reduction do.

Inputs, batch 1:

• Synthetic scan (generated by us, no dataset, Apache-2.0):

  • made by research/frnet/scripts/frnet_preproc_ref.py synthetic_scan (ray cast against ground, cars, facades, poles and pedestrians; seed 0; sensor 2 m above the ground) through logs/frnet/m5_baseline/scripts/make_bench_inputs.py. Its fp32 CPU reference (ONNX Runtime on the pinned deployed ONNX) sits next to it;
  • OT128: 512 columns, 53,411 points, 52,613 after the ego crop, no interpolated point, N = M = 52,613;
  • QT128: 600 columns, 45,783 points, 41,806 after the crop, N = 69,553 after interpolation, M = 18,926;
  • it stands in for the release's synthetic shipped sample (the same generator), which is not built yet. Update at the release: the shipped code/tt_frnet/samples/synthetic_<variant>.npz (code/scripts/make_sample.py) are byte-identical to these bench inputs, so the synthetic rows below are the shipped sample's.

• PandaSet 019 frame 40 (CC BY 4.0 + PandaSet terms), today's code/tt_frnet/samples/pandaset_019_40.npz:

  • 107,146 points, 106,501 after the ego crop;
  • OT128: N = 145,961 after interpolation, M = 40,691; QT128: N = 106,585, M = 6,255.

  The first release moved it to the git-ignored staging_samples_pandaset/samples/ (not shipped until PandaSet samples are approved); the commands below name its old path.

Configuration of every number, unless a row says otherwise:

• ETH dispatch, 1 CQ, 12×10 grid (device.compute_with_storage_grid_size(), printed in each log);
• the shipped precision knobs of tt-model.yaml serve.env: FRNET_POINT_TERMS=2 (two-term bf16 point island), FRNET_MAP_DTYPE=float32 (fp32 image-branch maps), FRNET_CONV_TERMS=1. The job unsets every other FRNET_*, so the code defaults (= the serve.env pins) apply.

Environment

item	value
tt-metal	44d66500520 (main) + patches/tt-metal-eth-dispatch.patch (sha256 08d0ddf6…, the same file as the workspace copy)
ttaw	0.20.0, vendored from common 89dec49 (code/tt_frnet/ttaw/VENDORED.json, source_dirty: false). vendor.py --check: 0 differences, so no re-vendor was needed
device	Blackhole p150b, KMD 2.10.0. ETH dispatch, 1 CQ, 12×10 = 120 cores, l1_small_size 64 KiB, trace_region_size 256 MiB (8.7 MiB used by OT128, 5.6 MiB by QT128). One trace per process; 227 programs in the program cache (OT128), 174 (QT128)
AICLK	1350 MHz median (min 1343) over 4,568 (OT128) and 3,278 (QT128) samples, one every 50 ms, during the stage benches (ttaw.profiling.AiclkSampler, sysfs); the same in every matrix run. Board power median 78 W (max 209 W), ASIC 64.9–71.7 °C. The profiler CSV header reads CHIP_FREQ 1350 MHz
numerics	point island (encoder, compression, point stem / fusions / fuse, head): fp32 activations, every linear as three bf16 matmuls (x_hi W_hi + x_hi W_lo + x_lo W_hi) with fp32 outputs summed in fp32, HiFi4. Image branch: fp32 weights, bf16 conv operands, fp32 maps (convs write fp32; HardSwish, adds and addcmul in fp32), HiFi4 + fp32 dest acc + packer L1 acc. Gathers and resize outputs bf16. The profile shows every matmul and conv as HiFi4
host	AMD EPYC-Rome VM, 8 vCPUs, 90 GB RAM, Linux 6.8. Shared with other agents' jobs: load average 4.1–6.6 during the device job. OMP_NUM_THREADS=4. Host rows move with that load; device rows do not (b2b rounds repeat to 0.02 ms)

How to run

ROOT=/home/ubuntu/experiments/tt-models; cd $ROOT/bundles/frnet-p150; source $ROOT/bin/tt-env.sh
export PYTHONPATH=$PWD/code:$PYTHONPATH HF_HUB_OFFLINE=1 OMP_NUM_THREADS=4 \
       FRNET_WEIGHTS_DIR=$ROOT/assets/frnet/hf_lidar_frnet          # v2.0, sha256-pinned
# accuracy gates (per-module PCC + end-to-end agreement vs the fp32 CPU reference, both profiles), one job
$ROOT/bin/devrun -t 1800 -- env TTAW_GATES_READONLY=1 python -m pytest -p no:cacheprovider -q -s \
    code/tt_frnet/tests/test_pcc_device.py code/tt_frnet/tests/test_e2e_device.py
# stage breakdown (pre split, host staging, H2D split, trace, b2b, D2H split, unsort, post, e2e p50/p99), with a check
# of each served output against its stored CPU reference (<stem>.<variant>.reference.json next to the input)
$ROOT/bin/devrun -t 900 -- python code/scripts/bench.py --variant ot128 --iters 100 --warmup 10 --b2b-iters 50 \
    --b2b-rounds 5 --input <synthetic_ot128.npz> --input code/tt_frnet/samples/pandaset_019_40.npz --json <out>.json
# dispatch / CQ matrix: one process per configuration
$ROOT/bin/devrun -t 900 -- python code/scripts/bench.py --variant qt128 --iters 50 --input <npz> --dispatch worker --cqs 2 --json <out>.json
# device profile: one eager forward (module / layer signposts) + one traced replay between signposts (1,104 programs:
# above the profiler's default 1,000-op buffer, hence --op-support-count; each section is flushed on its own)
$ROOT/bin/devrun -t 3600 -- python -m tracy -r -p -v --no-web-server --op-support-count 4000 \
    -o $ROOT/generated/profiler/frnet_baseline code/scripts/profile_ops.py --variant ot128 --input <npz>
tt-perf-report <ops_perf_results_*.csv> --start-signpost trace --end-signpost trace_end --tracing-mode
python code/scripts/profile_summary.py <ops_perf_results_*.csv> profile_summary.json --md profile_summary.md
# ttnn-visualizer memory + graph report of one eager forward (slow: never used for timing; the 7.5 GB graph JSON goes to tmpfs)
$ROOT/bin/devrun -t 2400 -- python code/scripts/visualizer_report.py $ROOT/generated/ttnn_visualizer/frnet_baseline \
    --variant ot128 --input code/tt_frnet/samples/pandaset_019_40.npz --json-dir /dev/shm/frnet_visualizer
ttnn-visualizer --profiler-path $ROOT/generated/ttnn_visualizer/frnet_baseline \
    --performance-path $ROOT/generated/profiler/frnet_baseline/reports/2026_10_09_13_56_53

• All device measurements ran in two bin/devrun windows on 2026-10-09, every step rc 0 (job script logs/frnet/m5_baseline/scripts/job1_all.sh, environment env.sh):

  • 13:08:29–13:10:07 UTC: the device suite (step 1);
  • 13:26:44–14:04:39 UTC: stage benches, matrix, both profiles and the visualizer report (steps 2–5).

  The first window also started steps 2–5, which failed in 1 s each: the HF cache holds no snapshot of the pinned revision, so env.sh now sets FRNET_WEIGHTS_DIR (logs of that attempt: logs/frnet/m5_baseline/try0/).

• The OT128 replay is 1,104 programs, above the device profiler's default buffer of 1,000. The run raises the buffer with --op-support-count 4000, and profile_ops.py flushes each section with ttnn.ReadDeviceProfiler. The replay's op sequence matches the eager one op code by op code (1,104 / 1,104; QT128 759 / 759), so no op was dropped.

Accuracy (baseline)

The device suite ran read-only (TTAW_GATES_READONLY=1, reports tagged _m5: logs/frnet/{pcc,e2e,latency}_{ot128,qt128}_m5.json, logs/frnet/m5_baseline/try0/suite_r1.log): 12 passed in 94.5 s. The values are bit-identical to PORT_LOG M4 (j6) and VERIFY_PORT round 1 (the graph is deterministic). The gate JSONs are unchanged (git status clean).

check	gate	OT128	QT128
every gated module tap (PCC vs the fp32 reference, sample, replay outputs)	≥ 0.999	min 0.999729 (point_fusion1)	min 0.999649 (attention4)
pred_probs PCC / raw-point argmax agreement, sample	≥ 0.998 / ≥ 98.5 %	0.999942 / 99.581 %	0.999844 / 99.295 %
public frames (OT128 17, QT128 5): min pred_probs PCC	≥ 0.998	0.999688	0.999805
public frames: min raw-point argmax agreement	≥ 98.5 %	98.907 % (nuScenes 0916 kf40; margin 0.41 pt)	99.158 % (PandaSet 029 f40)
replay == eager	bit-exact	45 / 45 taps	45 / 45 taps
CPU reference vs ONNX Runtime on the deployed ONNX (the reference itself is right)	≥ 0.9999 per tap	≥ 0.99999999999 (VERIFY_PORT, research venv)	same

• Every bench run also checks its served output against the stored fp32 CPU reference with the container-smoke gates (server/smoke_test.py REFERENCE_GATES); all passed:
  • OT128: synthetic labels 99.55 % / filter 99.77 % (max |Δscore| 0.072); PandaSet 99.59 % / 99.76 % (0.109);
  • QT128: synthetic 99.62 % / 99.87 % (0.144); PandaSet 99.29 % / 99.62 % (0.150);
  • every matrix run (ETH / WORKER × 1 / 2 CQs, both profiles) passes the same check.
• The binding constraint of the optimization phase is nuScenes 0916 kf40 (OT128): 98.907 % against the 98.5 % gate. Every precision-reducing change measured in the port failed it: bf16 maps 98.28 %, an fp32 × fp32 point island 97.50 % (PORT_LOG M3). See "The precision cost of the policy" below.
• Host suite on the fake ttnn with the new scripts: 112 passed, 13 skipped (the device files and the onnxruntime-only test). Log: logs/frnet/m5_baseline/host_suite.log.

Performance (warm, batch 1, median of 100 iterations)

Each stage is synchronised on its own, so the rows do not add up to e2e exactly. Values are p50 (p99) in ms; b2b is the median of 5 rounds of 50 back-to-back replays. pre includes the file decode (pre.load); an HTTP request decodes its points envelope inside the call.

stage	OT128 synthetic	OT128 PandaSet 019 f40	QT128 synthetic	QT128 PandaSet 019 f40
host preprocess as served (model._prepare)	55.3 (59.1)	107.2 (120.0)	54.2 (69.7)	83.5 (106.1)
… decode (load_points)	6.9	12.9	6.2	13.1
… ego crop / projection / unique cells	1.6 / 3.4 / 1.8	3.2 / 8.0 / 2.4	1.5 / 4.7 / 0.9	3.3 / 5.9 / 1.0
… fp16 interpolation grid + interpolated points	18.4	37.4	17.2	29.3
… device plan (build_plan: sort, CSR offsets, gather tables, 8-d features)	22.5	41.8	23.3	31.4
FrameStaging.write (range checks + persistent host buffers)	3.3	3.2	2.4	2.3
H2D, 9 inputs (8.6 MB OT128, 6.2 MB QT128)	5.42 (5.57)	5.41 (5.65)	4.04 (5.59)	4.00 (4.48)
… of which features fp32 [N_cap, 8]	4.13	4.13	3.14	3.13
device trace, one blocking forward	210.02 (210.15)	211.72 (211.89)	133.53 (134.01)	134.91 (135.12)
back-to-back trace replays, per forward	209.948	211.643	133.434	134.866
D2H as served (fp32 pred_probs [N_cap, 32] → numpy)	7.65 (20.2)	7.43 (19.8)	5.80 (8.7)	5.55 (7.6)
… copy / host untilize (to numpy)	2.25 / 4.95	2.29 / 5.01	1.77 / 4.15	1.78 / 3.75
unsort (first N rows / 27 columns, network-input order)	7.1 (14.1)	12.8 (23.3)	6.1 (13.2)	5.9 (13.7)
host postprocess (first-max argmax, filter, output assembly)	5.0 (6.8)	14.8 (19.3)	6.8 (9.8)	10.5 (15.9)
e2e model()	292.6 (299.4)	355.7 (367.0)	221.1 (233.5)	244.3 (258.7)

• Throughput.
  • Back-to-back replays give 4.72 frames/s of device throughput on OT128 (PandaSet) and 7.41 frames/s on QT128.
  • A synchronous model() call gives 2.81 frames/s (OT128 PandaSet), 3.42 (OT128 synthetic), 4.09 (QT128 PandaSet) and 4.52 (QT128 synthetic).
• The API's own timing_ms medians (OT128 PandaSet): preprocess 105.4 ms, device 235.7 ms, postprocess 14.6 ms, total 355.7 ms. The API's device stage is staging + H2D + trace + D2H + unsort: 24 of its 236 ms are outside the replay.
• The device is 60 % of e2e on OT128 (211.6 of 355.7 ms, PandaSet) and 55 % on QT128 (134.9 of 244.3 ms). The host preprocess is 30 % (OT128) and 34 % (QT128).
• The device time barely depends on the frame (fixed N_cap / M_cap): the synthetic OT128 scan with 36 % of PandaSet's points replays only 1.7 ms faster (209.95 vs 211.64 ms). The difference is in K1 and the gathers.
• Load (warm JIT cache): from_pretrained 6.5 s OT128 / 6.9 s QT128 in the stage bench (build 0.28 s, warm-up + capture 2.0 s). WORKER (11×10) compiles its own program set: 137 s OT128, 82 s QT128. A process with the device profiler compiles its own kernels (146 s).

Dispatch / CQ matrix (D14)

Setup:

• synthetic scan, one process per configuration, all in one devrun job (13:34–13:53 UTC);
• 50 iterations plus 5 × 30 back-to-back replays each;
• host load 4.1–5.8 throughout.

profile	configuration	grid	programs	b2b replay ms	trace p50 ms	H2D p50 / min ms	D2H p50 / min ms	e2e p50 / min ms
OT128	ETH-1CQ (shipped)	12×10	227	209.948	210.00	5.39 / 5.35	7.61 / 7.33	292.9 / 291.1
OT128	ETH-2CQ	12×10	227	211.317	211.45	5.49 / 5.46	7.65 / 7.30	290.5 / 289.7
OT128	WORKER-2CQ	11×10	231	211.567	211.73	4.57 / 4.55	7.56 / 7.19	293.4 / 291.7
OT128	WORKER-1CQ	11×10	231	211.563	211.63	4.49 / 4.47	7.48 / 7.32	291.7 / 291.2
OT128	ETH-1CQ again (drift check)	12×10	227	209.959	210.02	5.40 / 5.34	7.46 / 7.17	291.7 / 290.8
QT128	ETH-1CQ (shipped)	12×10	174	133.437	133.51	4.02 / 3.98	5.51 / 5.33	208.8 / 207.5
QT128	ETH-2CQ	12×10	174	134.250	134.38	4.11 / 4.08	5.50 / 5.28	210.0 / 209.1
QT128	WORKER-2CQ	11×10	174	153.721	153.82	3.45 / 3.42	5.53 / 5.31	229.3 / 227.7
QT128	WORKER-1CQ	11×10	174	153.709	153.77	3.39 / 3.37	5.43 / 5.30	226.0 / 225.3
QT128	ETH-1CQ again (drift check)	12×10	174	133.433	133.50	4.01 / 3.98	5.42 / 5.31	206.3 / 205.5

• ETH stays the default: it is better on the device in both profiles.
  • OT128: ETH replays 1.6 ms faster than WORKER (209.95 vs 211.56 ms, −0.8 %).
  • QT128: ETH replays 20.3 ms faster (133.44 vs 153.71 ms, −13.2 %). On the 128 × 256 grid the 11×10 partitioning of the point matmuls and convs is clearly worse. This is the largest ETH/WORKER gap measured so far in this workspace.
  • WORKER's H2D is 0.6–0.9 ms faster, as in the other matrices. It does not offset the replay.
• 1 CQ. ETH with 2 CQs replays 0.8–1.4 ms slower (ETH-2CQ runs without dispatch_s, PLAN §0.2). A synchronous request gains nothing from the second queue (e2e within host noise). Uploads are 5.4 ms and readbacks 7.5 ms per frame, so pipelined 2CQ staging could hide at most about 13 ms of a 212 ms frame (−6 %).
  • Decision: ETH, 1 CQ stays pinned (serve.env FRNET_DISPATCH=eth, FRNET_NUM_CQS=1, D14).
  • Re-measure a pipelined 2CQ mode once the replay is no longer 15–30× the transfers.
• The drift checks repeat to 0.01 ms (OT128) and 0.004 ms (QT128). One OT128 ETH-1CQ b2b round read 211.76 ms (the other four 209.94–209.95), and the median is unaffected.

Host vs device, and every host<->device transfer per forward

Host. Only the documented fallbacks of PLAN §2.8 run on the host; there is no host round trip and no host fallback op inside the device graph.

Before the device (host/preprocess.py, numpy), in the node's order:

• the ego crop (float32 R p + t);
• the float32 projection;
• the fp16 farthest-point interpolation grid and the interpolated points;
• the unique cells and the profile checks;
• the device plan: the sort by cell, CSR offsets, point2pix_s / pix2vox_s for s = 1, 2, 4, 8, and the 8-d features with pre_norm folded.

After the readback:

• the un-permute (plan.unsort) of the first N rows / 27 columns;
• the first-max argmax with the −1e9 start (never a device last-dim argmax, P10), the score, the class filter (prob ≥ 0.05 on drivable_surface) and the output assembly.

H2D, 9 per frame. copy_host_to_device_tensor from the persistent HostStaging buffers into the trace inputs, all ROW_MAJOR in DRAM (OT128 bytes; QT128 in brackets):

input	shape, dtype	bytes	ms
features	[1, 1, N_cap, 8] fp32 (32 B pages)	5,120,000 (3,840,000)	4.13 (3.13)
offsets (K1 CSR)	[1, 60002] uint32	240,008	0.17
p2p_1/2/4/8 (pixel2point)	4 × [1, N_cap] uint32	4 × 640,000 (480,000)	0.27–0.31 each
v2p_1/2/4/8 (frustum2pixel)	[1, (H/s)(W/s)] uint32	524,288 + 131,072 + 32,768 + 8,192	0.06–0.26
total		8,616,328 (6,174,088)	5.41 (4.00)

features moves at 1.24 GB/s: 160,000 pages of 32 bytes. The other inputs reach 1.4–2.1 GB/s.

D2H, 1 per frame. The fp32 pred_probs TILE [N_cap, 32]:

• 20,480,000 B (OT128) / 15,360,000 B (QT128);
• of the 32 columns, 27 hold classes and 5 are padding (bias −1e30: probability 0);
• rows beyond N are padding (PandaSet OT128 uses 145,961 of 160,000, the nuScenes frames 20–28 k);
• copy 2.29 ms (8.9 GB/s), then host untilize 5.0 ms.

The served device path (the API's device stage) is 236 ms on OT128 PandaSet. 212 ms of it run on the device.

Grid usage

• Nothing hard-codes 11×10 or 12×10 (VERIFY_PORT §3(b): the grid comes from device.compute_with_storage_grid_size(); K1 splits its work on the device by two binary searches, so ETH and WORKER give the same outputs). On 11×10, WORKER builds 231 instead of 227 programs (OT128).

• OT128: 456 of the 1,104 programs per frame run on all 120 cores, 648 on fewer (26.3 ms of kernel time):

  • every point-island op on N_cap rows, the K1 kernels and the softmax run on 120 cores. The M_cap-row compression matmuls run on 118;
  • the image-branch convs run on 96 (28 conv programs), 108 (22) and 40 cores (stage 4, pixel_fusion4, attention4: the 16 × 128 stride-8 map). Their glue (halo, move, padded slice, slice write, resharding) runs on 40–114 cores. The height-sharded configurations use whole row groups;
  • the convs reach a weighted 59.7 % of the HiFi4 FLOPs (tt-perf-report). The ten block-sharded conv programs reach 26.1 %, but take only 0.42 ms.

• QT128: the point matmuls run on 118 cores (3,750 tile rows of 120,000 points). The convs of the narrow 128 × 256 grid use 32–108 cores.

• L1 is empty between ops. The ttnn-visualizer report shows no L1 buffer at any op boundary.

  • Every activation round-trips DRAM interleaved.
  • DRAM peaks at 218.6 MB per bank (1.75 GB in all, weights included) at the first partial-sum add of decoder.point_fuse1. At that point the fp32 point features p0–p4 ([160000, 128], 82 MB each) are still alive next to the fp32 [160000, 256] partials (164 MB each).
  • L1 for tensors is 167.5 MB (120 × 1,395,712 B, the visualizer's device record).

• What would fit in L1:

  • a 128 × 1024 × 128 image map: 33.5 MB in bf16, 67 MB in fp32;
  • every point-island weight: the largest, point_fuse1 640 × 256, is 655 KB as two bf16 terms.

  No fp32 point tensor at N_cap fits (82–164 MB each), so the point island needs row-block streaming.

What is already fused / traced

Done:

• One trace per process covers the whole device graph: encoder → K1 → compression → frustum2pixel → stem → point stem → fusion stem → 4 stages → decoder → point fuse → head → softmax → one packed output.
  • Capture takes 2.6 s (OT128) / 2.0 s (QT128).
  • No eager op runs between replays; replay == eager bit for bit (a gate).
  • Eagerly, under the profiler, the same 1,104 programs span 260.2 ms (op-to-op gaps 47.6 ms of host dispatch), against 213.3 ms traced.
• K1 segment_reduce (owned ttnn.generic_op, bit-exact). It replaces the log-step stock-op fallback, which would take 3R + 1 programs per site, with one program per segmented-max site (6 per frame). It reads the matmul's TILE output directly (no untilize). Sites 1–5 write the bf16 frustum2pixel gather table, zero row included.
• Exact rewrites (all from the port, each tested against ORT or an fp32 emulation):
  • BN folded into the 14 point linears and the 53 convs;
  • pre_norm folded into the host features;
  • frustum2pixel / pixel2point as TILE + PADDED ttnn.embedding gathers with a zero sentinel row (C19);
  • concat-free point layers (K-split linears) and K-split fusion convs. fuse_layer1 is a 5 × 128 split-K KSplitConv (P14), so the 160 MiB decoder concat never exists;
  • align-corners resizes as interpolation matmuls (C18);
  • the 27 logits padded to 32 with a −1e30 bias, so the softmax is exact;
  • fused SIGMOID in the attention convs, ReLU in the point linears;
  • a bf16 input of a two-term linear (the gathered map rows) takes two matmuls instead of three (lo = 0).
• Host side: HostStaging writes the plan into persistent zero-copy host tensors (no per-frame from_torch), and out-of-profile frames return before touching the device.

Not fused / not resident:

• the two-term point island:

  • each point linear runs one split per fp32 input: typecast to bf16, typecast back, subtract, typecast. So 4 programs per input;
  • then 2–3 bf16 matmuls per part, each writing a full fp32 [N_cap, C_out] partial to DRAM;
  • then one fp32 add per partial.

  The 5-part point_fuse1 alone is 49 programs and 39.9 ms;

• every activation in DRAM interleaved between ops. The 128 × 1024 convs run ttnn's automatic DRAM slicing, each slice with its own padded slice, halo, move and slice write;

• every HardSwish is its own fp32 ttnn.hardswish (P5: a fused HardSwish is silently skipped), 46 per frame;

• the resizes (C18) run as transpose + matmul + reshape + tilize / untilize chains with bf16 outputs (C18 cannot write fp32 rows at 1024 × 128);

• pixel2point: untilize of the map, then the gather, then a typecast to fp32;

• every point-wise op runs on all N_cap rows and K1 writes all M_cap segments, whatever the frame's N and M;

• no megakernel. Projection, interpolation, the plan, the argmax and the filter run on the host.

Device profile (traced replay; kernel sum vs span, op-to-op gaps)

One traced replay, ETH-1CQ, PandaSet 019 f40. Reports: OT128 generated/profiler/frnet_baseline/reports/2026_10_09_13_56_53/, QT128 generated/profiler/frnet_baseline_qt128/reports/2026_10_09_13_59_21/.

	OT128	QT128
programs per frame	1,104	759
distinct programs (program cache)	227	174
ttnn calls per frame (visualizer, OT128)	491: 123 typecast, 84 add, 72 linear, 62 conv2d, 46 hardswish, 27 to_layout, 21 reshape, 20 subtract, 12 embedding, 6 generic_op (K1), 6 transpose, 6 matmul (resize), 4 addcmul, 1 pad, 1 softmax; plus 439 deallocate	–
device kernel sum	212.64 ms	134.81 ms
span (first FW start → last FW end at 1350 MHz)	213.27 ms (b2b 211.64)	n/a (see below; b2b 134.87)
op-to-op gaps	0.628 ms in all, median 0.55 µs	0.443 ms, median 0.57 µs
conv + matmul kernels	76.70 ms (36 %)	51.62 ms (38 %)
math fidelity	HiFi4 on all 677 compute ops	HiFi4 on all 539

• The trace is kernel-time bound: dispatch is not a sink (0.3 % of the span). Cutting programs pays only through the kernel time of the programs it removes. A megakernel's dispatch saving is at most 0.6 ms.
• QT128's span is not usable: one replayed conv (row 3631 of the CSV, backbone.layer4) reports a FW duration of 3.18 s against a 41 µs kernel, a profiler artefact. The b2b replay (134.87 ms, five rounds within 0.01 ms) shows no such stall. The kernel sum plus gaps (135.25 ms) is the QT128 reference.

By part of the graph (logs/frnet/m5_baseline/scripts/group_profile.py; ops attributed through the eager module / layer signposts):

part	OT128 ops	OT128 kernel ms	share	QT128 ms	share	of which (OT128)
point island (encoder ffe0–3, compression, point stem, point fusion 1–4, point fuse, head + softmax)	215	133.66	62.9 %	101.12	75.0 %	matmul 61.2, fp32 adds 52.6, typecasts (two-term splits) 19.5
… encoder ffe0–3 (8 → 64 → 128 → 256 → 256)	36	22.14	10.4 %	16.71	12.4 %	ffe3 9.9 (three 1.6 ms matmuls)
… point stem + point fusion 1–4 (K-split 384 / 256 → 128)	65	37.88	17.8 %	28.36	21.0 %	7.0 per fusion, 10.0 the stem
… point fuse (640 → 256 → 128)	58	47.40	22.3 %	35.77	26.5 %	point_fuse1 39.9: 15 matmuls 17.4, 15 adds 16.4, 20 split ops 6.1
… head (4 × linear + skips, logits, softmax)	44	23.89	11.2 %	17.95	13.3 %
… compression (256 → 16, on M_cap rows)	12	2.35	1.1 %	2.33	1.7 %
K1 segment max (6 sites) + tilize	7	17.22	8.1 %	15.18	11.3 %	encoder site 4.93 (fp32 × 256), 5 × 2.39 (× 128)
pixel2point gathers (6 × untilize + embedding + typecast)	18	6.15	2.9 %	3.50	2.6 %	embedding 3.47
frustum2pixel gathers (6 embeddings)	6	0.65	0.3 %	0.15	0.1 %
decoder resizes (C18, 3 maps)	39	9.03	4.2 %	2.24	1.7 %	matmul 2.89, reshape 2.62, (un)tilize 2.30
image branch: stem, fusion stem, stages 1–4 (convs + glue)	605	30.66	14.4 %	8.63	6.4 %	conv 7.62, fp32 HardSwish 5.25, typecast 4.24, DRAM slicing 5.19, adds 3.21, halo 1.96
image branch: decoder fuse1 (K-split 5 × 128 → 256) + fuse2	214	15.26	7.2 %	3.98	3.0 %	conv 4.96, K-split adds 3.62, slicing 3.89
total	1,104	212.64		134.81

sink (op type, OT128)	µs	%	where
Matmul (78)	64,118	30.2	72 point-island matmuls at 1.4–7.9 % of the HiFi4 FLOPs and 130–200 GB/s (26–39 % of the DRAM peak): each writes a full fp32 partial [160000, C_out] (82–164 MB). The ffe3 parts take 1.6 ms each, a point_fuse1 part 1.16 ms, a point-fusion part 0.67 ms. Plus 6 resize matmuls (2.9 ms)
BinaryNg (104)	59,441	28.0	point island 52.6 ms: 59 fp32 adds (the two-term / K-split partial sums and the head's skips; 1.09 ms each at [160000, 256], about 450 GB/s: DRAM-bound) and the 20 subtracts of the two-term splits. Image branch 6.8 ms: fp32 residual adds and the decoder's K-split sums (3.6 ms)
Typecast (123)	26,795	12.6	61 fp32 ↔ bf16 casts of the two-term splits (19.5 ms); 53 casts of the fp32 maps to bf16 before the convs (5.2 ms); 6 pixel2point casts to fp32 (1.5 ms); 3 in the resizes (0.7 ms)
GenericOp: K1 (6)	16,903	7.9	scalar RISC-V segment max. The encoder site reads 164 MB in 4.93 ms (33 GB/s); each 128-channel site takes 2.39 ms. It scales with M_cap (60,000 output rows each) more than with the rows: QT128 runs the same sites in 2.0–4.4 ms on 75 % of the rows
Conv2d (138)	12,582	5.9	53 convs in 2–4 DRAM slices each, weighted 59.7 % of the HiFi4 FLOPs
SliceWrite + PaddedSlice (194)	9,073	4.3	ttnn's DRAM slicing of the 128 × 1024 convs
Unary (46)	6,269	2.9	the 46 fp32 HardSwish
Embedding (12)	4,118	1.9	pixel2point (3.47 ms) and frustum2pixel (0.65 ms) gathers
Halo + Move + resharding (358)	4,816	2.3	conv glue
Reshape + (Un)Tilize + Transpose (39)	7,340	3.5	resize chains, pixel2point untilize, the input tilize
Ternary addcmul (4)	852	0.4	attention gating

Top 10 device ops of the OT128 replay:

#	op	where	µs	cores
1	K1 segment_reduce	segmax.encoder: [160000, 256] fp32 → [60000, 256] fp32 (voxel max)	4,928	120
2–6	K1 segment_reduce	segmax.points sites 1–5: [160000, 128] → the bf16 gather table [60001, 128]	2,394–2,396 each	120
7–9	Matmul	encoder.ffe3 (256 → 256), the three two-term products [160000, 256] × [256, 256] → fp32	1,641 / 1,599 / 1,598	120
10	Matmul	head.mlp2 part [160000, 256] × [256, 128] → fp32	1,221	120

Next come the 15 point_fuse1 matmuls (1.15–1.20 ms each), the 25 fp32 adds at [160000, 256] (1.08–1.11 ms each) and the point_stem part matmuls (1.19 ms).

Most expensive layers:

• decoder.point_fuse1: 39.85 ms (18.7 % of the replay), of which 17.41 ms matmul;
• decoder.fuse1: 12.55 ms in 157 programs, of which 3.92 ms conv;
• K1 segmax.points (5 sites): 11.98 ms;
• backbone.point_stem: 9.96 ms;
• encoder.ffe3: 9.88 ms;
• head.mlp1 / head.mlp2: 8.23 / 8.09 ms;
• each point fusion: 6.97–6.99 ms;
• stage-1 convs: 1.43–1.90 ms each, of which 0.37 ms compute (the rest is the DRAM slicing, halo and the fp32 output).

logs/frnet/m5_baseline/profile_summary_{ot128,qt128}.{json,md} have the per-module, per-layer and per-op tables, tt_perf_report_{ot128,qt128}.txt the tt-perf-report output (with its per-op advice), and grp_{ot128,qt128}.txt the part breakdown above.

The precision cost of the policy. These are the port's same-session device A/Bs on the OT128 frames (PORT_LOG M1–M3, jobs j2–j5, gates read-only). They were not re-run for this baseline.

policy	replay ms	Δ vs shipped	OT128 frames min argmax (nuScenes 0916 kf40)	gate
fp32 × fp32 point island (POINT_TERMS=1), bf16 maps (j2)	107	−105 ms (−50 %)	97.50 %	fails
two-term point island, bf16 maps (j3)	201	−11 ms (−5 %)	98.28 %	fails
two-term point island, bf16 maps, CONV_TERMS=2 (j4)	244	+32 ms	98.35 %	fails
two-term point island, fp32 maps (shipped; j5 / this baseline)	212	–	98.91 %	passes, 0.41 pt margin
two-term point island, fp32 maps fed to the convs directly (j5)	217	+5 ms	98.13 %	fails
two-term point island, fp32 maps, CONV_TERMS=2 (j5)	249	+37 ms	99.00 %	passes, 0.50 pt margin

• The two-term point island costs about 94 ms per OT128 frame (j2 → j3: 107 → 201 ms, +88 %). That is 44 % of today's replay, and it is how the port reaches the gate: the device's fp32 matmul is TF32-like (P12). The profile shows that the cost is not the extra products (bf16 HiFi4 is exact) but the fp32 partial tensors, their DRAM adds and the split typecasts. Item 1 below targets exactly that, with the same products.
• The fp32 maps cost about 11 ms (201 → 212 ms): fp32 HardSwish, residual adds and addcmul, and a cast to bf16 before every conv. They buy 0.63 pt on the binding frame.
• Cheaper precision fails. Every cheaper policy measured fails the binding nuScenes frame. HiFi2 / LoFi or bfp8 were not measured: with a 0.41 pt margin and 98.28 % for bf16 maps, they need fresh device evidence, and the FPU is not the bottleneck (the matmuls run at 1.4–7.9 % of the FLOPs).
• More margin costs time. CONV_TERMS=2 raises the margin to 0.50 pt for +37 ms. fp32 resize outputs (blocked by C18 at 1024 × 128 rows) and fp32 gathers for the 128-channel maps are unmeasured options.

Ranked optimization opportunities

How to read the estimates:

• Gains are on the OT128 trace (211.6 ms, PandaSet) unless marked e2e (OT128 PandaSet e2e 355.7 ms p50). QT128 follows the same ranking (the point island is 75 % of its 134.9 ms).
• Each estimate is the measured kernel or host time of the work the item removes, with a range for what replaces it. Items overlap, so the gains do not add up.
• Every item keeps the frozen gates and the precision policy (PLAN §9.3), and lands as one commit with one FRNET_* A/B knob (PLAN §5.1).
• Judge numerics changes on the binding frame first: nuScenes 0916 kf40, OT128, 98.907 % against the 98.5 % gate. A change in the order of fp32 accumulation is not bit-identical to today, so items 1–2 must re-run the full device suite.

1. Two-term point linears as one K-concatenated matmul per layer (trace −60..−80 ms, −28..−38 %; the largest item).
   • Today each point linear writes one fp32 [N_cap, C_out] partial per product: 2–3 per input part, up to 15 for point_fuse1. One fp32 add per partial then sums them. Matmuls take 61.2 ms and the adds 52.6 ms, at 130–450 GB/s: DRAM traffic, not FLOPs (1.4–7.9 % of HiFi4).
   • The fix: one matmul per layer over the K-concatenated bf16 operands [x_hi | x_hi | x_lo] (all parts) × [W_hi; W_lo; W_hi], with fp32 dest accumulation and packer L1 accumulation.
     • The products are the same exact bf16 products. Only the fp32 summation order changes, so the gates must be re-run.
     • The fp32 output is written once. The bf16 inputs are a quarter of an fp32 partial's bytes.
   • The concat of the bf16 operands costs about 0.3–1 ms per layer, unless the split kernel (item 3) writes them straight into the K-slices of one tensor.
   • Expected result: matmul about 61 → 25–35 ms, adds 52.6 → about 5 ms (the residual / skip adds stay).
2. MK-PointMLP: a row-block-streamed point MLP megakernel (trace −100..−115 ms beyond today, overlapping 1; D19).

   • The point island is 133.7 ms (OT128) and 101.1 ms (QT128): 63–75 % of the replay. Its work is row-local except the gathers and K1.

   • A kernel per row block (for example 32–64 points per core) chains the linears of one segment in L1:

     • the encoder ffe0..ffe3;
     • point fuse 640 → 256 → 128;
     • the head's 4 layers + skips + logits + softmax.

     It does the two-term splits on the fly, holds the bf16 weight terms in L1 (at most 655 KB for point_fuse1) and writes only the segment's outputs. The head segment writes pred_probs directly.

   • Its DRAM floor is a few tens of MB per frame, about 0.1–0.3 ms. Compute at the measured bf16 rates fits in roughly 15–30 ms for the whole island.

   • Stepping stones:

     • item 1;
     • then fuse each linear's split into the producer's epilogue (item 3);
     • then one segment at a time, the head first (23.9 ms, no gather inside).
3. A fused two-term split (trace −10..−14 ms; on the way to 1–2).
   • Each fp32 input today runs 4 programs: typecast, typecast back, subtract, typecast. That is 19.5 ms for 61 casts, plus 20 subtracts (in the BinaryNg row), at N_cap.
   • One generic_op (or SFPU eltwise) pass can read fp32 and write hi and lo bf16, about 0.3–0.5 ms per [160000, 256] input. It can write into the K-slices of item 1's operand. Exact: the same RNE casts.
4. N_cap / M_cap buckets (trace −100..−160 ms on frames with N ≤ 32–64 k; e.g. every nuScenes frame).
   • Every point-wise op runs on all 160,000 rows (OT128) or 120,000 (QT128), and K1 writes all 60,000 segments. That covers the point island, K1, the gathers and the readback: about 157 ms of the OT128 replay.
   • The catalogued nuScenes frames have N = 20–28 k (13–17 % of N_cap) and M = 17–25 k. The QT128 PandaSet frame has M = 6,255 against M_cap = 60,000.
   • The fix: two or three trace variants per profile (for example N_cap 32 k / 64 k / 160 k, M_cap 16 k / 60 k), sharing weights. The host picks the smallest bucket that covers the frame. Each variant costs about 6–9 MiB of trace region (256 MiB available) plus its activations.
   • PandaSet-like frames (N = 146 k) gain nothing. The readback shrinks with N too (item 9).
5. K1 throughput (trace −10..−14 ms; owned shared kernel).
   • K1 takes 16.9 ms: 33 GB/s on the 164 MB encoder site, and 2.0–2.4 ms per 128-channel site almost independent of the row count. So per-segment output overhead dominates: 60,000 segments of one row each through an 8-row staging ring.
   • Fixes, in order:
     • batch output rows per worker (contiguous segments → one NOC write per block);
     • move the accumulator from L1 into the RISC-V's local memory (PORT_LOG known issue);
     • with item 4, write only M_cap-bucket segments;
     • a compute-engine reduction is the long form.
   • Bit-exactness must stay (max of integer keys). The shared-kernel rule (PLAN §5.1.5) applies: re-vendor and re-run K1's host / device tests.
6. Image branch DRAM slicing and glue (trace −8..−14 ms; MK-ImageStage).
   • The 128 × 1024 convs spend 12.6 ms in conv kernels but 9.1 ms in DRAM slicing (padded slice + slice write) and 4.8 ms in halo / move / resharding. The fp32-map glue adds the following:
     • 46 fp32 HardSwish: 6.3 ms;
     • the casts to bf16 before each conv: 5.2 ms;
     • fp32 residual adds and K-split sums: 6.8 ms.
   • A 128 × 1024 × 128 map is 33.5 MB in bf16 / 67 MB in fp32 against 167.5 MB of L1. So the stride-1 stage chain can stay height-sharded in L1 (C17 slicing="l1"), which drops the slicing.
   • Fold the bf16 cast of the conv input and the HardSwish into one eltwise pass (exact: the same ops).
   • The longer form is a BasicBlock megakernel with halo exchange (MK-ImageStage).
7. Decoder resizes and fuse1 (trace −5..−8 ms).
   • Resizes take 9.0 ms: reshape 2.6, tilize / untilize 2.3, transpose 0.4 and matmul 2.9.
   • decoder.fuse1 takes 12.5 ms in 157 programs: a 5-part K-split over five 128 × 1024 maps, with 3.6 ms of K-split adds and 3.9 ms of slicing.
   • Fix: run the resize matmuls in TILE layout end to end, so no reshape / (un)tilize is needed. That also lifts the C18 fp32 limitation, a precision-margin option. Accumulate the K-split parts in the conv (dest / L1 accumulation) instead of fp32 adds.
8. pixel2point gathers (trace −2..−4 ms).
   • Today: untilize of each map (1.2 ms), the bf16 embedding (3.5 ms), a typecast to fp32 (1.5 ms).
   • The two-term linear already treats a bf16 input as exact (lo = 0). So feed the gather output as bf16 straight into item 1's operand (no cast). Exact.
   • Later: gather from the TILE map in the MK-PointMLP reader.
9. Host preprocess (e2e −60..−85 ms OT128).
   • Today 107 ms on PandaSet:
     • plan 41.8 ms: sort, CSR, gather tables, features;
     • fp16 interpolation 37.4 ms;
     • decode 12.9 ms;
     • projection 8.0 ms;
     • crop and unique cells 5.6 ms.
   • The fixes:
     • a C++ / numba implementation of the node's CUDA kernels (projection + interpolation grid in one pass, counting-sort plan): about 10–20 ms in all;
     • keep the decode outside the timed request path.
   • The numerics must stay bit-identical to today's host code (float32 / fp16 policies, the documented race policies). The host tests and the research goldens check that bit for bit.
10. Host after the device (e2e −15..−25 ms OT128).
    • Today: unsort 12.8 ms + post 14.8 ms + host untilize 5.0 ms.
    • Fixes: read back row-major (device untilize, or a ROW_MAJOR output), with only N rows (item 4). Fuse the un-permute into the argmax pass: take the argmax over the 27 columns on the permuted rows, then scatter int8 labels and float scores instead of the 27-column float rows.
    • Exact: the same first-max rule on the same fp32 values.
11. H2D features layout (e2e −3.5 ms). features moves 5.1 MB at 1.24 GB/s in 160,000 pages of 32 bytes. Upload it as [N_cap / 4, 32] rows (128 B pages) and view or reshape on the device. A TILE upload would also drop the input tilize (0.36 ms).
12. Throughput by pipelining (frames/s; latency unchanged).
    • Device b2b allows 4.72 frames/s (OT128) and 7.41 (QT128). A synchronous request gives 2.81 / 4.09 frames/s on PandaSet.
    • Overlap the host pre / post of frames k ± 1 with the device work of frame k in the server.
    • 2CQ staging can hide the 5.4 ms upload (and, with double-buffered outputs, the 7.4 ms readback). It is worth measuring only after items 1–2 shrink the replay.
13. Megakernel candidates (D19, PLAN §5.2 / §5.3).
    • The §5.2 gate is met through DRAM round trips and layout glue, not dispatch. 64 % of the replay is not conv / matmul compute, while op-to-op gaps are 0.3 %.
    • Candidates, by measured block time:
      • MK-PointMLP (point island, 133.7 ms OT128 / 101.1 ms QT128; items 1–3, 8): first, by far. It is row-local, and its weights fit L1;
      • MK-SegGather (K1 + the frustum2pixel / pixel2point gathers, 24 ms; items 5, 8): fold the gather tables into K1's writer and the pixel2point gather into the point reader;
      • MK-ImageStage (stem + stages 1–4 + decoder convs, 46 ms, of which 12.6 ms conv compute; items 6–7): band-fused BasicBlocks with halo exchange. The 128 × 1024 maps fit L1 in bf16.
    • A single whole-model kernel is not practical: the fp32 point tensors at N_cap (82–164 MB each) do not fit L1. MK-PointMLP's row-block streaming is the practical "single" form for 63–75 % of the work.
14. Precision margin (accuracy, not speed; optimization-phase precision item).
    • The margin on nuScenes 0916 kf40 is 0.41 pt. Items 1–2 change the fp32 accumulation order, so re-check the margin after each step.
    • Options to buy margin, measured: CONV_TERMS=2 +0.09 pt for +37 ms. Unmeasured: fp32 resize outputs (after item 7) and fp32 gathers of the 128-channel maps (ttnn.embedding is bf16-only; a K1-style generic_op gather).
    • No precision relaxation (bf16 maps, a TF32 point island, HiFi2) is a candidate without new device evidence on that frame: each one measured so far fails it.

Raw logs and files

All under /home/ubuntu/experiments/tt-models/:

• logs/frnet/m5_baseline/job1.log (+ job1.rcs): the device job (steps 2–5). try0/: the first attempt (suite log suite_r1.log; the other steps failed on the weights path). job1_prereboot.log: the job queued before the server reboot, which never started;
• logs/frnet/m5_baseline/bench_eth1cq_{ot128,qt128}.{json,log}: the stage benches;
• logs/frnet/m5_baseline/matrix_{ot128,qt128}_{eth-1cq,eth-2cq,worker-2cq,worker-1cq,eth-1cq-repeat}.{json,log};
• logs/frnet/m5_baseline/profile_{ot128,qt128}.log, profile_summary_{ot128,qt128}.{json,md}, tt_perf_report_{ot128,qt128}.txt, grp_{ot128,qt128}.txt;
• logs/frnet/m5_baseline/visualizer.log, host_suite.log; logs/frnet/{pcc,pcc_small,e2e,latency}_{ot128,qt128}_m5.json;
• logs/frnet/m5_baseline/inputs/: the synthetic scans, their calibrations and CPU references;
• device profiles: generated/profiler/frnet_baseline/reports/2026_10_09_13_56_53/ops_perf_results_2026_10_09_13_56_53.csv (OT128) and generated/profiler/frnet_baseline_qt128/reports/2026_10_09_13_59_21/ops_perf_results_2026_10_09_13_59_21.csv (QT128);
• ttnn-visualizer report: generated/ttnn_visualizer/frnet_baseline/ (db.sqlite 26 MB, config.json; OT128 PandaSet, one eager forward. The 7.5 GB graph JSON went to /dev/shm and was deleted after the import).