# meteor-p150 baseline on the p150 (before optimization)

Date 2026-10-10. Code is the baseline commit `e86c13a` (first correct port, no optimization; ETH default with the
hang fix of PORT_LOG.md section 12). The model code of `code/tt_meteor/` is unchanged since then
(`git diff e86c13a -- code/tt_meteor` is empty); this file's commit adds only the measurement scripts
(`code/scripts/bench.py` rewritten with p50 / p99, several inputs, AICLK and the agreement check; `profile_ops.py`;
`profile_summary.py`). tt-metal `44d66500520` + `patches/tt-metal-eth-dispatch.patch` +
`patches/tt-metal-reshape-rm-sys1419.patch`; ttaw 0.23.2. Weights `AutowareFoundation/meteor` @ `01a5f6d71df`
(tag `v1.0`), `meteor_v157c3Z.onnx`, run as released (`input_norm=onnx`, D12).

Inputs, batch 1 (one frame = 8 camera slots of 768×432 + calibration + ego speed):
- **sample**: `code/tt_meteor/samples/pandaset_019_f40.json`, frame 40 of PandaSet sequence 019 in METEOR's
  8-slot layout (7 camera JPEGs + the absent BACK_NARROW; preset `pandaset_019`; ego speed 10.88 m/s; CC BY 4.0 +
  PandaSet Dataset Terms). It is the sample the port ships today. The release may move it to the git-ignored
  `staging_samples_pandaset/` until the user approves shipping PandaSet data; `bench.py` finds it there too. Its
  stored fp32 CPU reference gives the agreement check of every run;
- **synthetic**: eight 768×432 uint8 noise images made by `bench.py` (data generated by us), preset `pandaset_019`,
  10 m/s. The kind of input a release can always ship;
- **PandaSet 090 f40** and **nuScenes 0103 kf09**: public-dataset frames of two other rigs, as API requests built
  from `research/meteor/public_data/inputs` (local only, never shipped; nuScenes is CC BY-NC-SA). Their camera
  arrays, K, T and v0 equal the goldens' graph inputs exactly (checked on the host before the bench).

Configuration of every number: ETH dispatch, 1 CQ, 12×10 grid (`device.compute_with_storage_grid_size()` printed in
each log), the shipped precision policy (PLAN §9.3: fp32 weights, HiFi4 + fp32 accumulation; image branch and det
path in three bf16 terms, `METEOR_IMAGE_PRECISION=terms3`; BEV trunk bf16 activations), unless a row says otherwise.

## Environment

| item | value |
|---|---|
| tt-metal | `44d66500520` (main) + `patches/tt-metal-eth-dispatch.patch` (sha256 `08d0ddf6…`) + `patches/tt-metal-reshape-rm-sys1419.patch` (sha256 `74878683…`, the SYS-1419 row-major reshape fix that ended the ETH hangs, PORT_LOG 12.7-12.8). Both are applied to the workspace tree (`git diff --stat`: 5 files) and shipped in `patches/`; `eth_patch: true` in every log |
| ttaw | 0.23.2, vendored from `common` `60b6dd7` (`code/tt_meteor/ttaw/VENDORED.json`; `vendor.py --check`: 0 differences, so no re-vendor and no gate re-run was needed) |
| device | Blackhole p150b, KMD 2.10.0, firmware bundle 19.13.1.0. ETH dispatch, 1 CQ, 12×10 = 120 cores, `l1_small_size` 32 KiB, `trace_region_size` 256 MiB. ONE trace (`frame`, the whole network), **4,678 programs per frame**, 1,131 programs in the program cache |
| AICLK | 1350 MHz median (min 1343) over 8,412 samples (one every 50 ms) during every bench and matrix loop (`ttaw.profiling.AiclkSampler`, sysfs); board power median 69-78 W (max 203 W), ASIC 64.5-71.3 °C. The profiler CSV header reads CHIP_FREQ 1350 MHz |
| numerics | HiFi4 on all 2,402 compute programs of a frame (profile) |
| host | AMD EPYC-Rome VM, 8 vCPUs, 90 GB RAM, Linux 6.8, shared with up to 8 other agents' jobs: load average 4.1-12.5 during the stage bench, 4.5-12.1 during the matrix; `OMP_NUM_THREADS=4`. Host rows move with that load (the sample's first plain loop ran at load 12.5 and has the worst e2e p99); device rows do not (trace p50 556.7-556.9 ms on every input and run) |

## How to run

```bash
ROOT=/home/ubuntu/experiments/tt-models; cd $ROOT/bundles/meteor-p150; source $ROOT/bin/tt-env.sh
export PYTHONPATH=$PWD/code:$PYTHONPATH HF_HUB_OFFLINE=1 TT_MODEL_WEIGHTS_REVISION=01a5f6d71df5ecbbb5853ec600825481d57b9c6b
# (every METEOR device job needs METEOR_DEVICE_OK=1 from the orchestrator; never arm tt-triage)
# accuracy gates (per-module PCC + end-to-end agreement vs the fp32 CPU reference), read-only, one job
$ROOT/bin/devrun -t 1800 -- env TTAW_GATES_READONLY=1 python -m pytest -q -s \
    code/tt_meteor/tests/test_pcc_device.py code/tt_meteor/tests/test_e2e_device.py
# stage breakdown (API split, host_in / H2D / trace / b2b / D2H / unpack, e2e p50 / p99, AICLK, agreement)
$ROOT/bin/devrun -t 2400 -- python code/scripts/bench.py --iters 60 --inputs sample,synthetic,<name>=<manifest> --json <out>.json
# dispatch / CQ matrix: one process per configuration
$ROOT/bin/devrun -t 2700 -- python code/scripts/bench.py --inputs sample --iters 50 --dispatch worker --num-cqs 2 --json <out>.json
# rig switch (each call writes the other rig's lift tables)
$ROOT/bin/devrun -t 1500 -- python code/scripts/bench.py --inputs a=<rig A manifest>,b=<rig B manifest> --iters 5 --rig-switch 40 --json <out>.json
# device profile: one eager frame with module signposts + two signposted traced replays
$ROOT/bin/devrun -t 3600 -- python -m tracy -r -p -v --no-web-server --op-support-count 6000 \
    -o $ROOT/generated/profiler/meteor_baseline code/scripts/profile_ops.py --rounds 2
tt-perf-report <ops_perf_results_*.csv> --start-signpost frame --end-signpost frame_end --tracing-mode --arch blackhole
python code/scripts/profile_summary.py $ROOT/generated/profiler/meteor_baseline --json s.json --md s.md
# ttnn-visualizer graph report of one eager frame (slow, never used for timing)
$ROOT/bin/devrun -t 2700 -- python code/scripts/profile_ops.py --graph-report $ROOT/generated/ttnn_reports/meteor_baseline
ttnn-visualizer --profiler-path $ROOT/generated/ttnn_reports/meteor_baseline/frame \
    --performance-path $ROOT/generated/profiler/meteor_baseline/reports/2026_10_10_15_12_27
```

- **One frame is 4,678 programs**, above the device profiler's default buffer of about 1,000 programs, so the
  profile runs with `--op-support-count 6000` and flushes the buffer (`ttnn.ReadDeviceProfiler`) before and after
  each section.
  - The log warns of dropped markers at 15:05:53-15:06:02. That is the flush right after the compile frame (the
    first eager frame, 806.6 s of JIT with the profiler build), before any measured window.
  - The measured windows are complete: both traced rounds have 4,678 programs and kernel sums of 555.28 / 555.61 ms,
    and the eager frame has the same op sequence as the trace, op for op (4,678 programs, kernel sum 554.86 ms).
- The job drivers and analysis scripts are in `logs/meteor/baseline/scripts/` of the workspace (`env.sh` holds the
  common environment, `run_rest.sh` / `run_e.sh` / `run_f.sh` the job order).

## Accuracy (baseline)

The model code, ttaw and weights are those of `e86c13a`, whose gates ran three times under ETH on 2026-10-10
(PORT_LOG 12.8, `logs/meteor/hangfix/suite_final{1,2,3}.log`: 15 passed each, read-only gates, identical values in
all three runs). `vendor.py --check` reports 0 differences and `code/tt_meteor/` is unchanged, so the frozen gates
were not re-run for this file.

| check | gate | baseline (device suite x3 under ETH, read-only gates: 15 passed each, identical values) |
|---|---|---|
| teacher-forced stage PCC (26 gates of `tests/test_pcc_device.py`, PandaSet 019 f40) | >= 0.99 | min 0.9999890 (`tf.risk`); `depth_mean` 0.99948 (reported) |
| teacher-forced lane / seg2d / depth argmax agreement | >= 0.99 | 0.9984 / 0.9965 / 0.9936 |
| chained frame (the served graph from the cameras), min output PCC: PandaSet 019 / 090, nuScenes 0103, valday #40 | >= 0.99 | 0.99989 / 0.99990 / 0.99991 / 0.99982 |
| chained lane / seg2d / depth agreement (worst frame) | >= 0.99 | 0.9963 / 0.9965 / 0.9925 |
| det3d strict recall / precision (served decode): sample, PandaSet 090, nuScenes, valday #40 | >= 0.95 / >= 0.95 | 1.0/1.0, 1.0/1.0, 1.0/1.0, **0.971 / 0.971** (34 / 35) |
| selected ego path deviation (max waypoint), worst frame | <= 0.3 m | 0.170 m (PandaSet 090) |
| sample `/predict` vs stored CPU-reference body (smoke compare): recall / precision / max \|Δscore\| | >= 0.95 / >= 0.95 / - | 1.0 / 1.0 / 0.0081 |
| imagenet + linear variant (`test_variants_device.py`), float outputs min PCC | >= 0.99 | 0.99988 |
| CPU reference vs ONNX Runtime on the shipped ONNX (the reference itself is right) | >= 0.9999 | min 0.99999990 (155 taps) |
| ETH stress (`code/scripts/stress_frames.py --mode rigs`: rig change every frame, bit-compare per rig) | no hang, 0 mismatches | 2000 frames clean |

- **Every bench run also checks its own served output.** The sample runs as a fresh stream and is compared with
  its stored CPU reference with the container-smoke gates (`ttaw.server.smoke.compare_with_reference`):
  8 / 8 boxes matched, recall = precision = 1.0, max |Δscore| 0.0081, lane agreement 0.99770, selected path
  deviation 0.108 m. Identical in ETH-1CQ, ETH-2CQ and WORKER-2CQ (WORKER lane agreement 0.99770 too).
- Host suite on the fake ttnn (the host test files, explicitly listed): **100 passed, 1 skipped**
  (`test_reference_cpu.py`: no onnxruntime in tt-env), `logs/meteor/baseline/host_suite.log`.

## Performance (warm, batch 1, p50 (p99) of 60 iterations per loop)

Each row of the API split is timed inside one `model(...)` call (wrappers around the parts of
`TtMETEOR.run_frame`); the `TraceRunner` rows come from a second loop with every stage synchronised on its own.
Images are decoded before timing (the camera driver hands METEOR decoded images).

| stage | sample (PandaSet 019) | synthetic | PandaSet 090 | nuScenes 0103 |
|---|---:|---:|---:|---:|
| host preprocess (`_prepare`: INTER_AREA to 768×432, K / T, METEOR's runtime pre-processing) | 11.2 (35.8) | 11.1 (13.2) | 9.3 (10.3) | 9.2 (10.1) |
| device stage of the API (`api.device`), of which: | 774.9 (883.9) | 735.1 (756.6) | 755.0 (787.2) | 765.9 (780.8) |
| … NCHW → NHWC camera rows on the host (`image_rows`) | 26.3 (31.9) | 23.1 (30.9) | 21.6 (32.0) | 22.1 (35.6) |
| … `TraceRunner` call: host tensor + H2D + replay + segmented D2H | 689.5 (711.4) | 646.6 (668.8) | 677.7 (694.4) | 675.9 (700.1) |
| … join of the 5 readback segments (`unpack`) | 15.9 (53.6) | 15.6 (16.1) | 15.6 (16.5) | 15.6 (16.4) |
| … device layouts → the 19 ONNX outputs (`outputs_from_device`, host) | 49.4 (95.9) | 47.8 (51.8) | 46.0 (52.4) | 45.7 (50.9) |
| host postprocess (METEOR's C++ decode rules + temporal state) | 129.9 (164.6) | 124.0 (130.7) | 130.0 (141.3) | 131.2 (143.1) |
| **e2e `model()`** | **918.2** (1071.8) | **875.7** (901.8) | **902.8** (927.8) | **914.4** (931.1) |
| `TraceRunner` stages: host tensor (`host_in`) | 1.17 (1.39) | 1.14 (1.22) | 1.17 (3.11) | 1.12 (1.17) |
| H2D (8 × 432 × 768 × 3 uint8 + v0, synchronised) | 66.5 (74.2) | 66.4 (84.2) | 66.4 (68.2) | 66.4 (67.6) |
| **device trace, one blocking replay** | **556.85** (559.34) | 556.87 (562.78) | 556.85 (559.24) | 556.68 (556.97) |
| **back-to-back replays, per frame** (20) | **556.75** | 556.79 | 556.82 | 556.57 |
| D2H (5 segments, 75.0 MB fp32) | 17.2 (63.5) | 17.8 (66.7) | 47.5 (70.5) | 46.7 (67.8) |
| `TraceRunner` e2e (call + unpack) | 658.3 (680.5) | 658.2 (670.2) | 688.3 (720.2) | 688.1 (708.4) |

- **Throughput.** Back-to-back replays give **1.80 frames/s** of device throughput (556.8 ms per frame); a
  synchronous `model()` call gives 1.09-1.14 frames/s (875.7-918.2 ms).
- **The device time does not depend on the input** (one fixed-shape graph; the lift tables only change the
  sampling positions): trace p50 556.7-556.9 ms on all four inputs, AICLK 1350 MHz throughout.
- **Where the time goes on the sample (918 ms):** device trace 557 ms (61 %); host decode 130 ms (14 %); host
  conversion of the 75 MB readback (`unpack` + `outputs_from_device`) 65 ms (7 %); H2D 66 ms (7 %); host
  NCHW → NHWC 26 ms; D2H 17-48 ms; host preprocess 11 ms.
- **D2H is bimodal** (17 ms or 46-48 ms for the same 75 MB, both within one process; p99 63-71 ms on every input).
  The device and the bytes are the same, so it is a host-side effect of the shared host (cause not isolated;
  UNVERIFIED: page faults or memory pressure while other agents run).
- **Load** with a warm JIT cache: 49.7 s (weights 0.3 s, graph build 2.6 s, warm-up 42.2 s: one eager frame 41.0 s
  plus the capture 1.2 s). A process on a new dispatch configuration compiles its own kernels: ETH-2CQ 141.9 s,
  WORKER-2CQ (11×10) 767.7 s.
- **Rig change** (`--rig-switch`; calls alternate between two calibrations, so every call writes the other rig's
  `lift_grid` + 20 table chunks): PandaSet 090 ↔ nuScenes 0103, 40 calls (`rigswitch_eth-1cq.json`): the table
  write takes **182.3 ms p50** (p99 203.9; 211 MB in 21 synchronised transfers, about 1.16 GB/s) and the
  call 1128.5 ms p50 (p99 1203.1) instead of about 905 ms. The lift geometry itself is cached per rig on the host. A
  fixed rig (the vehicle case) pays this once. (The stage bench's own `--rig-switch` loop alternated the sample and
  the synthetic input, which share the `pandaset_019` preset, so it measured no rig change; it is not quoted.)

## Dispatch / CQ matrix (D14)

Setup: the sample (identical frames in every row), one process per configuration, each its own devrun window
(14:29-14:51 UTC), 50 iterations per loop plus 20 back-to-back frames. The first row is the stage bench.

| configuration | grid | programs (cache) | b2b frame ms | trace p50 (p99) ms | H2D p50 / min ms | D2H p50 ms | runner e2e p50 ms | e2e p50 (p99) / min ms | host load |
|---|---|---:|---:|---:|---:|---:|---:|---:|---|
| **ETH-1CQ** (shipped; stage bench) | 12×10 | 1,131 | **556.75** | **556.85** (559.34) | 66.46 / 66.42 | 17.19 | 658.3 | 918.2 (1071.8) / 883.1 | 12.5 → 5.4 |
| ETH-2CQ | 12×10 | 1,131 | 561.37 | 561.52 (562.05) | 67.36 / 67.33 | 18.02 | 662.4 | 916.8 (941.8) / 881.4 | 5.1 → 4.5 |
| WORKER-2CQ | 11×10 | 1,161 | 578.85 | 579.04 (579.46) | 62.81 / 62.76 | 17.30 | 675.8 | 965.9 (1086.0) / 907.6 | 12.1 → 10.6 |
| WORKER-1CQ (PORT_LOG 12.6, measured at an earlier commit, for reference only) | 11×10 | | | 573.0 | | | | | |

- **ETH stays the default.** WORKER-2CQ replays the frame 22.1 ms slower (578.9 vs 556.8 ms, +4.0 %): on 11×10
  ttnn partitions differently and builds 1,161 programs instead of 1,131. WORKER's H2D is 3.6 ms faster (62.8 vs
  66.4 ms), which does not make up the difference. WORKER also needs its own 13-minute JIT compile at first load.
- **1 CQ.** ETH-2CQ replays 4.6 ms slower than ETH-1CQ (561.4 vs 556.8 ms; ETH-2CQ runs without dispatch_s, PLAN
  §0.2), and a synchronous request gains nothing from the second queue (e2e 916.8 vs 918.2 ms at a lower host load).
  **Decision: 1 CQ stays pinned** (`DEVICE_DEFAULTS num_command_queues: 1`, `METEOR_NUM_CQS`, D14): the API and the
  server serve one synchronous request at a time. A pipelined serve mode (opportunity 8) is the only way 2 CQs pay.
- The ETH-2CQ and WORKER-2CQ processes ran clean (rc 0, outputs identical to ETH-1CQ on the agreement check). Before
  the SYS-1419 reshape fix, ETH-2CQ hung within 43 frames (PORT_LOG 12.5); with the fix it is a working option again.

## Host vs device, and every host<->device transfer per forward

**Host** (METEOR's own runtimes define pre / post: there is no Autoware package):
- before the device: request parsing, the INTER_AREA resize to 768×432 (identity on the 768×432 sample), K scaled
  per axis, T inverted, the absent narrow cameras as zero images with their donor's pose (9-11 ms); the lift geometry
  and tables per calibration (cached per rig); the NCHW → NHWC transpose of the 8 cameras (22-26 ms);
- after the readback: the join of the segments (15.6 ms), `outputs_from_device` (device layouts → the 19 ONNX
  outputs: views, transposes, uint32 → uint8, 46-49 ms), and METEOR's C++ decode rules
  (`host.result.build_output`, 124-131 ms).

There is no mid-graph host round trip and no host fallback op: the whole network is ONE trace, replayed per call.

**H2D per frame** (`copy_host_to_device_tensor` into the persistent trace inputs, CQ0):

| tensor | shape, dtype, layout | bytes | pages × page size | when |
|---|---|---:|---|---|
| `imgs` | `[1, 1, 2654208, 3]` uint8 ROW_MAJOR (8 × 432 × 768 NHWC rows) | 7,962,624 | **2,654,208 × 3 B** | every frame: **66.4 ms (0.12 GB/s)** |
| `v0` (RT-dev param) | `[1, 1, 1, 1]` fp32 TILE | 4 KB (one tile) | | when the speed changes (unchanged values are not re-uploaded) |
| `lift_grid` | `[8, 400, 250, 2]` fp32 ROW_MAJOR | 6,400,000 | | when the rig changes |
| `lift_table_00..19` | 20 × `[1, 1, 40000, 64]` fp32 TILE | 20 × 10,240,000 | | when the rig changes; every chunk is followed by a device sync |

**D2H per frame:** one packed fp32 tensor `[18309, 1024]` (75.0 MB: `pack_outputs` of the 15 device outputs, the
argmaxes as uint32 → fp32) read as 5 row segments of at most 16.8 MB, 17.2 ms (4.4 GB/s) when the host is calm.

**The image upload is the transfer problem.** 8 MB goes up as 2.65 M pages of 3 B (one pixel per page): 0.12 GB/s.
The same path moves the 75 MB readback at 4.4 GB/s with 4 KB pages, so camera rows as pages (2,304 B) would take
about 2-3 ms (UNVERIFIED estimate from the measured D2H rate).

## Grid usage

- **Nothing hard-codes 11×10 or 12×10** (VERIFY_PORT): on 11×10 the WORKER run builds its own program set (1,161 vs
  1,131) and gives the same served output.
- **1,916 of the 4,678 programs run on all 120 cores, and they hold 407.6 of the 555.3 ms** (73 %):
  the DRAM-sliced convs and their slices, typecasts, the binary ops, the TILE reshapes, the lift grid_samples.
- 1,234 programs on 110-119 cores (65.8 ms): convs and their halo / move on height-sharded slices.
- 1,198 programs on 64-109 cores (51.0 ms), 220 on 16-63 cores (18.3 ms), 110 on 1-15 cores (12.6 ms). The costly
  few-core programs:
  - `head.ego_attn` pooling matmul `[1, 800, 96, 500] × [500, 16]` fp32 on **3 cores: 5.10 ms**;
  - `head.risk_sample` `[1, 400000] × [400000, 96]` bf16 (a K = 400,000 reduction) on **3 cores: 4.81 ms**;
  - the W-axis interpolation matmuls of `Resize2d` at 250 → 500 on 16-24 cores: `lift.resize` 2.96 ms,
    `head.seg_refine` 2.89 ms, `bev.lane` 2.89 ms; `bev.traj` `[400·128, 250] × [250, 16]` on 4 cores 1.48 ms.
- **Every activation is DRAM-interleaved between ops** (C17 / C27 first-port layout). The convs run ttnn's
  automatic DRAM slicing with height-sharded L1 slices (386 PaddedSlice + 386 SliceWrite programs).
- **L1 is empty between ops.** The ttnn-visualizer graph report of one eager frame (2,820 recorded ttnn operations,
  37,001 buffer records, all DRAM) shows no L1 buffer at any op boundary. DRAM peaks at 95.9 MiB per bank at the
  lift's fp32 product `[8, 1, 100000, 96] × w` (the 307 MB fp32 camera products next to the 205 MB of fp32 lift
  tables, the 8-camera fp32 feature maps of the three-term image branch and the 48.25 M weights).

## What is already fused / traced

**Done:**
- **The whole network is one trace** (`frame`, 4,678 programs), replayed per call with no host work inside. Capture
  takes 1.2 s; replay == eager is a gate of the port (M7).
- **Eagerly** (profiler, program cache hot), the same 4,678 programs take the same kernel time (554.86 vs 555.28 ms
  traced), but the eager frame spans 806.1 ms: **251.3 ms of op-to-op gaps** (host dispatch of 4,678 ops, the most
  before `img.depth` 67.4 ms and `img.resnet` 55.3 ms). Traced, the gaps add up to 2.74 ms (0.5 %).
- **Exact rewrites** (PORT_LOG section 2, all checked against the fp32 reference):
  - BN folded into every conv (float64); the /255 input scale folded into the stem weights; tgate + tfuse merged
    into one 1×1 conv (96 → 100); merged hm | reg heads (det and det2d); ctx + paint (21 → 96) as one K-split conv;
    the MHAs in folded form (attention pools); traj's `reg_pre` input folded in;
  - every Resize as separable fp32 interpolation matmuls (`F.interpolate` half-pixel incl. the edge clamp, also the
    non-integer ones);
  - the lift as two `grid_sample` gathers + a per-calibration bin-interpolation table + fp32 camera sums;
  - pools as constant matmuls, `mean(fused)` computed once;
  - argmaxes channel-major on the device; one packed readback in <= 16 MB segments.
- **Fused epilogues:** ReLU in every conv; fp32 conv outputs where a value feeds a threshold, a softmax, a clamp or
  the planner.

**Not fused / not resident:**
- every activation round-trips DRAM between ops;
- **the three-term convs (E20 / E22)**: every conv of the image branch and the det path runs three bf16 convs
  (`x_hi·W_hi`, `x_hi·W_lo`, `x_lo·W_hi`) plus a typecast pair to split the fp32 activation and two fp32 adds;
- **TILE reshape copies** between `[1, 1, N·H·W, C]` and `[1, H, W, C]` around every `Resize2d` and every conv
  whose map width (500, 250, 48, ...) is not a multiple of 32: 435 copies, 128.5 ms;
- the stem's exact fp32 max pool is a chain of DRAM untilize / pad / strided slices / maximums / tilize (25.3 ms);
- the output tail untilizes, pads and reshapes every output into the packed rows (12.9 ms);
- no custom kernel, no megakernel.

## Device profile (traced replay; kernel sum vs span, op-to-op gaps)

Traced replays, ETH-1CQ, the sample (`profile_ops.py --rounds 2`; stage labels from the eager frame's module
signposts, carried over op for op). `profile_summary.json` / `.md` hold every table below.

| | frame (round 1) | frame (round 2) |
|---|---:|---:|
| programs | **4,678** | 4,678 |
| device kernel sum | **555.28 ms** | 555.61 ms |
| span (first FW start → last FW end at 1350 MHz) | 558.02 ms (b2b 556.75) | 558.35 ms |
| op-to-op gaps | **2.74 ms (0.5 %)** | 2.74 ms |
| math fidelity | HiFi4 on all 2,402 compute programs | |

**The trace is kernel-time bound: dispatch is not a sink.** At 0.6 µs per op, removing every op-to-op gap would
save 2.7 ms; a megakernel pays only through the kernel time (DRAM round trips, layout copies) it removes.

**Compute vs everything else.** Data movement and layout ops, not math, dominate:
- TILE reshapes (23.1 %), typecasts (9.2 %), slicing (PaddedSlice / SliceWrite / Slice 13.1 %), tilize / untilize /
  pad (6.5 %), halo / move / sharding (3.6 %), transpose / concat / permute (1.3 %): **315 ms of the 555 ms (57 %)
  move or reformat data**;
- the convs (65.2 ms, 585 programs) run at 63 % of the HiFi4 FLOPs inside their kernels (height-sharded, weighted
  mean, tt-perf-report) and 31 % (block-sharded); the matmuls at 7.9 % (weighted mean);
- tt-perf-report models the frame at 2.1 % of the DRAM roofline (11 GB/s).

Per block and module (traced kernel time):

| block / module | ops | kernel ms | share | where the time goes (traced kernel ms) |
|---|---:|---:|---:|---|
| **image branch** (8 cameras, terms3) | **2,621** | **268.27** | **48.3 %** | |
| `img.normalize` (uint8 → bf16) | 1 | 8.57 | 1.5 % | one Typecast of `[2654208, 3]` ROW_MAJOR (3-byte pages) |
| `img.stem` (7×7 s2 conv + exact fp32 max pool) | 44 | 42.51 | 7.7 % | max pool chain 25.3 (strided Slice 17.56, maximums 4.29, pad 2.15, untilize 0.86, tilize 0.40), PaddedSlice of the 3-channel input 13.85 (6 DRAM slices, padded to 8 channels), the 6 conv slices 0.99 + their SliceWrites 1.22 |
| `img.resnet` (ResNet-34, terms3) | 949 | 57.54 | 10.4 % | BinaryNg 15.26 (term sums, residuals), conv 13.52, ReshapeView 10.41, Typecast 7.53 |
| `img.fpn` (laterals, resizes, adds) | 75 | 25.03 | 4.5 % | matmul 8.24 (resizes), ReshapeView 5.82, BinaryNg 4.72, untilize 2.28 |
| `img.fpn_fuse` | 96 | 7.66 | 1.4 % | |
| `img.seg2d` (decoder + clamp + argmax) | 373 | 40.89 | 7.4 % | BinaryNg 9.69, conv 6.95, Typecast 6.21, matmul 5.60 |
| `img.depth` (head + argmax + softmax) | 511 | 43.56 | 7.8 % | BinaryNg 12.25, conv 9.13, Typecast 7.07, SliceWrite 6.10, a `[64, 165888]` fp32 TILE reshape 3.34 |
| `img.depth_mean`, `img.ctx_paint`, `img.det2d`, `img.tl` (+ the lift-input conversions) | 572 | 42.51 | 7.7 % | `img.det2d` 28.12 (terms3) |
| **lift** | **61** | **56.26** | **10.1 %** | GridSample 28.21 (two calls of 14.1 ms: `[8, 108, 192, 96 / 64]` bf16 RM maps, `[8, 400, 250, 2]` fp32 grid → `[8, 400, 250, C]`), weights / sums BinaryNg 6.92, tilize 4.25, typecast 2.84; `lift.resize` ×2 10.58 (matmul 4.88, reshape 1.91) |
| **BEV trunk** (800×500 and 400×250 maps) | **939** | **147.48** | **26.6 %** | ReshapeView 69.88 (104 copies), conv 20.84, matmul 13.41, BinaryNg 11.48, SliceWrite 5.93 |
| … `bev.lane` | 326 | 58.16 | 10.5 % | ReshapeView 27.11, conv 12.65, matmul 7.52 |
| … `bev.det` (terms3 det stem + head) | 348 | 48.56 | 8.7 % | ReshapeView 24.00, BinaryNg 6.73, conv 5.20, typecast 3.76 |
| … `bev.fuse`, `bev.traj`, `bev.occupancy`, `bev.stationary`, `bev.risk` | 265 | 40.76 | 7.3 % | `bev.fuse` 17.13 (reshape 6.66), `bev.traj` 10.36 |
| **heads + planner** | **1,012** | **70.34** | **12.7 %** | ReshapeView 24.73, matmul 17.42, BinaryNg 5.32, conv 4.23 |
| … `head.seg_refine` | 264 | 34.19 | 6.2 % | ReshapeView 14.71, matmul 6.05 |
| … `head.box_refine` (terms3) | 476 | 15.69 | 2.8 % | ReshapeView 6.43, BinaryNg 3.06 |
| … planner (`ego_stem`, `ego_mlp`, `ego_attn`, `sem`, `risk_sample`, `decoder_wp`, `e2e`) | 272 | 20.46 | 3.7 % | `ego_attn` 8.09 (3-core matmul 5.10), `risk_sample` 5.07 (3-core matmul 4.81), `ego_stem` 4.38 |
| **output tail** (`out.pack`) | **45** | **12.93** | **2.3 %** | ReshapeView 6.00, pad 3.40, untilize 2.50 |
| **total** | **4,678** | **555.28** | | |

| sink (op type), frame | ms | % | where |
|---|---:|---:|---|
| ReshapeView (435) | 128.52 | 23.1 | TILE copies `[1, 1, N·H·W, C]` ↔ `[1, H, W, C]` at W = 500 / 250 / 48: `[400000, 96]` → `[800, 500, 96]` bf16 10 × 1.65 ms, `[100000, 128]` ↔ `[400, 250, 128]` (bf16 and fp32) 31 copies 18.8 ms, `[10368, 256]` ↔ `[216, 48, 256]` 90 copies 9.8 ms (resnet layer 3-4), ...; by module: `bev.lane` 27.1, `bev.det` 24.0, `head.seg_refine` 14.7, `img.resnet` 10.4, `bev.fuse` 6.7, `head.box_refine` 6.4, `out.pack` 6.0, `img.fpn` 5.8 |
| BinaryNg (440) | 85.66 | 15.4 | the fp32 term sums of the three-term convs, residual adds, the stem max-pool maximums, lift weights and camera sums |
| Conv2d (585) | 65.17 | 11.7 | each image / det conv three times (terms3); 63 % (height-sharded) / 31 % (block-sharded) of the HiFi4 FLOPs inside the kernels |
| Halo + Move + interleaved ↔ sharded (1,580) | 19.86 | 3.6 | the conv glue of the DRAM-sliced convs |
| Matmul (131) | 54.31 | 9.8 | the planner's two 3-core matmuls 9.91; the rest (44.4) mostly the `Resize2d` interpolation matmuls (fp32, many on 16-48 cores), pools and the paint fold |
| Typecast (359) | 50.91 | 9.2 | fp32 → bf16 TILE (234 programs, 25.68 ms) and bf16 → fp32 (121, 16.58 ms): the term split of the three-term convs; uint8 → bf16 input 8.57 |
| PaddedSlice + SliceWrite + Slice (820) | 72.56 | 13.1 | ttnn's automatic DRAM slicing of the convs 49.3 (the stem's 3-channel input 13.85); the max-pool strided slices 17.56; lift camera slices 2.08 |
| GridSample (3) | 28.22 | 5.1 | the lift (2 × 14.1 ms) and the planner's risk sampling |
| Tilize / Untilize / Pad (194) | 36.34 | 6.5 | the output tail 6.4, lift sampled-row tilize 4.25, the stem max pool 3.4, resize and planner glue |

Top 10 device ops of the frame (traced):

| # | op | where | µs | cores |
|---:|---|---|---:|---:|
| 1 | GridSample | `lift`: ctx `[8, 108, 192, 96]` bf16 RM, `[8, 400, 250, 2]` fp32 grid → `[8, 400, 250, 96]` | 14,119 | 120 |
| 2 | GridSample | `lift`: depth probabilities `[8, 108, 192, 64]` → `[8, 400, 250, 64]` | 14,093 | 120 |
| 3 | Typecast | `img.normalize`: `[2654208, 3]` uint8 → bf16 ROW_MAJOR | 8,575 | 120 |
| 4 | Matmul | `head.ego_attn`: `[1, 800, 96, 500] × [500, 16]` fp32 (attention pooling) | 5,098 | **3** |
| 5 | Matmul | `head.risk_sample`: `[1, 400000] × [400000, 96]` bf16 | 4,809 | **3** |
| 6 | Slice | `img.stem` max pool: `[8, 218, 386, 64]` fp32 RM → `[8, 218, 192, 64]` (stride 2) | 3,936 | 120 |
| 7 | Slice | `img.stem` max pool (second column phase) | 3,914 | 120 |
| 8 | Slice | `img.stem` max pool (third column phase) | 3,910 | 120 |
| 9 | ReshapeView | `img.depth`: `[1, 1, 64, 165888]` → `[1, 64, 5184, 32]` fp32 TILE | 3,335 | 120 |
| 10 | Matmul | `lift.resize`: `[1, 400, 96, 250] × [250, 500]` fp32 (W axis ×2) | 2,961 | 24 |

Then the two other W-axis resize matmuls at 250 → 500 (2,890 / 2,885 µs on 16 cores), the six PaddedSlices of the
stem's 3-channel input (2,285-2,361 µs each), the max-pool pad (2,150 µs) and the output tail's `[400384, 9]` →
`[3519, 1024]` reshape (2,086 µs).

- Shapes in the CSV come from the first program of a cached program (binary_ng reuses one program across shapes),
  so the per-module attribution uses the stage labels, not the CSV's shape columns. Ops a module issues inline after
  its last call belong to the module signposted last (e.g. the depth softmax to `img.depth_mean`, the lift-input
  conversions to `img.tl`).
- `tt_perf_report_frame.csv` has the per-op FLOPs and DRAM figures; `profile_extra.json` the per-module op-code
  tables, the reshape shapes and the core-count histogram.

**The precision cost found in the port** (A/B only: the bench's agreement check on the sample against its stored
CPU reference with the container-smoke gates; the frozen gates were not run):

| policy | b2b frame ms | Δ vs shipped | sample check (smoke gates): matched, recall / precision, max \|Δscore\|, lane agreement, path deviation | frozen gates |
|---|---:|---:|---|---|
| **shipped: image branch and det path in three bf16 terms (fp32 activations), BEV trunk bf16** | **556.75** | – | 8 / 8, 1.0 / 1.0, 0.0081, 0.99770, 0.108 m | all pass (above) |
| image branch with bf16 activations (`METEOR_IMAGE_PRECISION=bf16`; det path still terms3), measured here (`precision_image-bf16.json`, 20 iterations) | **390.83** | **−165.9 ms (−29.8 %)** | 8 / 8, 1.0 / 1.0, 0.0046, 0.99781, 0.077 m (center error up to 0.37 m) | **fails** the depth argmax gate: 0.9797 < 0.99 (PORT_LOG E20, port measurement on the same frame; not re-run here) |
| det path in bf16 (`det_precision="bf16"`, no env knob; port A/B, PORT_LOG 11.4: trace 548.5 vs 507.4 ms before the ETH / reshape changes) | | −41 ms (port) | | **fails** strict det3d on valday #40: recall 0.943 < 0.95 (PORT_LOG E22) |

- **The three-term precision of the image branch is the largest single cost of the port: 165.9 ms per frame
  (29.8 %)**, measured as a same-day A/B on the same build. Together with the det path (+41 ms at the port) the
  three-term policy accounts for about 207 ms (37 %). Both cheaper policies fail a frozen gate (depth argmax
  agreement, strict det3d recall), so the cost is real accuracy, not over-engineering. Item 2 targets the mechanism
  (three convs, a typecast split and two fp32 adds per conv), not the precision.
- The bf16 run passes the sample's smoke comparison, which shows how loose that check is next to the frozen gates.

## Ranked optimization opportunities

How to read the estimates:
- Gains are on the device frame (556.8 ms, 1.80 frames/s) unless marked e2e. Each one is the measured kernel time
  of the programs the item removes, with a range for what replaces them.
- Items overlap, so the gains do not add up.
- Every item keeps the frozen gates above and the precision policy (PLAN §9.3), and lands as one commit with one
  `METEOR_*` A/B knob (PLAN §5.1).

1. **TILE reshape copies (trace −90..−120 ms; the largest item).**
   - 435 `ReshapeView` programs take 128.5 ms (23.1 %). They are real copies because the map widths (500, 250,
     48, 125, ...) are not multiples of 32: `[1, 1, N·H·W, C]` ↔ `[1, H, W, C]` around every `Resize2d` and every
     conv whose neighbour wants the other form. By module: `bev.lane` 27.1, `bev.det` 24.0, `head.seg_refine` 14.7,
     `img.resnet` 10.4, `bev.fuse` 6.7, `head.box_refine` 6.4, `out.pack` 6.0, `img.fpn` 5.8 ms.
   - Fixes, one knob each: keep every FeatureMap in one form per stage (the convs take the flattened form; give
     `Resize2d` a flattened-input path); ROW_MAJOR views where a reshape is page-aligned (safe again under ETH with
     the SYS-1419 patch); or pad the BEV width to 512 / 256 so the TILE reshape is a view.
   - The same C17 / C24 item as in CenterPoint and BEVFormer (shared ttaw fix, re-vendor rule PLAN §5.1 step 5).
2. **Three-term convs as one fused op (trace −90..−150 ms; keeps the precision).**
   - Measured cost of the policy: 165.9 ms for the image branch (above) + about 41 ms for the det path.
   - Mechanism today, per conv: a typecast pair splits the fp32 activation into `x_hi` / `x_lo` (359 typecasts,
     50.9 ms per frame in total, of which 42.3 ms are TILE fp32 ↔ bf16 conversions, mostly these splits), three
     bf16 convs (585 convs, 65.2 ms), two fp32 adds (the image branch alone runs 282 BinaryNg programs, 61.9 ms).
   - A fused terms conv (one program: hi / lo split in the reader / unpacker, the three products accumulated in the
     fp32 dest, one output) removes the typecasts, the adds and two of three activation reads. Alternatively a true
     fp32-activation conv path of TF32+ quality (I7 investigates why ttnn's own fp32 path is worse).
   - Precision-preserving stepping stone: drop the `x_lo·W_hi` / `x_hi·W_lo` term per layer where the
     `precision_emulation.py` gates allow (E20 found two-term activations insufficient as a whole; per-layer is
     untested).
3. **Input path and stem (trace −35..−45 ms; e2e −60 ms).**
   - `img.normalize` 8.6 ms: a uint8 → bf16 typecast over 2.65 M ROW_MAJOR pages of 3 bytes.
   - The stem conv's DRAM slicing reads the 3-channel input six times and pads it to 8 channels (PaddedSlice
     13.9 ms).
   - The exact fp32 max pool is a DRAM chain (untilize, pad, 6 strided slices, 4 maximums, tilize): 25.3 ms. A
     generic_op 3×3 / s2 fp32 max pool on the conv's height-sharded output, or the pool folded into the stem
     output, takes it to about 1-3 ms (UNVERIFIED).
   - H2D: upload whole camera rows (2,304-byte pages) or a pre-padded 8-channel layout instead of 3-byte pages:
     66.4 ms → about 2-5 ms at the measured D2H rate (e2e).
4. **DRAM slicing and conv glue (trace −25..−40 ms).**
   - 386 PaddedSlice + 386 SliceWrite programs (49.3 ms) and the halo / move / sharding glue (19.9 ms) around the
     DRAM-sliced convs of the 8-camera image maps and the 800×500 BEV maps.
   - Larger slices, explicit height-sharded inputs (L1 holds 171 MB for tensors; layer-1 maps of 8 × 108 × 192 × 64
     fp32 are 42 MB), and the band streaming of item 9 (MK4).
5. **Lift as one gather kernel (K5 / MK3; trace −30..−45 ms).**
   - The lift takes 56.3 ms: two `grid_sample` calls of 14.1 ms that write `[8, 400, 250, C]` bf16 to DRAM
     (256 MB), then tilize, fp32 typecast, the product with the 205 MB fp32 table, sums over bins and cameras,
     divide, and the ×2 resize (10.6 ms).
   - A kernel per BEV block reading the L1-resident camera maps (ctx + prob: 53 MB bf16, fits the 171 MB of L1),
     with the per-calibration ELL weights (K slots) and fp32 accumulation, writes only `[400, 250, 96]`.
6. **Few-core and fp32 matmuls (trace −12..−20 ms).**
   - `head.ego_attn` `[800·96, 500] × [500, 16]` fp32 on 3 cores (5.10 ms) and `head.risk_sample` `[1, 400000] ×
     [400000, 96]` on 3 cores (4.81 ms): put the long dimension into M or split K over cores with a final
     reduction: well under 1 ms each.
   - The W-axis `Resize2d` matmuls at 250 → 500 run on 16-24 cores (2.9 ms each, 3 calls); `bev.traj`'s
     `[400·128, 250] × [250, 16]` on 4 cores (1.5 ms).
7. **Host side (e2e −150..−220 ms; latency only).**
   - Host decode 124-131 ms (`host_post_profile.json`, CPU-only cProfile on the golden outputs: 150 ms on the
     loaded host): the temporal seg fusion 73 ms (log-softmax of `[9, 800, 500]` 25 ms, contiguous copies 26 ms)
     and the 2D decode 69 ms (1,210 per-class `peak_mask` / `sigmoid32` calls on 8 cameras × 10 classes × 3
     scales). Vectorise the 2D peaks over classes and cameras and work in place: about −80..−100 ms (UNVERIFIED).
   - Readback conversion 61-65 ms (`unpack` 15.6 + `outputs_from_device` 46-49) and D2H 17-48 ms of 75 MB fp32:
     bf16 / uint8 outputs packed in their final layouts (views only) halve the bytes and remove most of the
     conversion: about −40..−60 ms.
   - The host NCHW → NHWC transpose (22-26 ms): pre-processing can write NHWC directly (the INTER_AREA output
     is HWC already): about −20 ms.
8. **Throughput by pipelining (frames/s; latency unchanged).**
   - The device allows 1.80 frames/s (556.8 ms b2b); a synchronous call gives 1.09-1.14 frames/s.
   - A pipelined serve mode (2 CQs: frame k+1's host pre-processing and upload, and frame k−1's readback and
     decode, overlapped with replay k) is bounded by max(device 561 ms on ETH-2CQ, host about 250-330 ms): about
     1.7-1.8 frames/s (UNMEASURED). It needs double-buffered outputs and reads on CQ1 (`TraceRunner` reads on CQ0
     today), and the per-stream host state must stay ordered.
9. **Megakernel and fusion candidates (D19, PLAN §5.2 / §5.3; SPEC §9 MK0-MK7).**
   - The §5.2 gate is met through DRAM round trips and layout copies (57 % of the kernel time), not dispatch:
     op-to-op gaps are 0.5 % of the frame (2.74 ms for 4,678 programs).
   - Candidates, by measured block time:
     - **MK4 band-streamed BEV trunk** (`bev.*` 147.5 ms + `head.seg_refine` / `box_refine` 49.9 ms = 197 ms, of
       which 91 ms are reshape copies): 800×500 bands in L1, no reshapes, no DRAM slicing. First, after item 1;
     - **MK1 / MK2 image encoder + heads** (268.3 ms, 48 %): L1-resident per camera group; item 2's fused terms conv
       is the precondition;
     - **MK3 lift** (56.3 ms; item 5);
     - **MK0 input** (normalize + stem slicing + max pool: 51 ms; item 3);
     - **MK6 planner** (20.5 ms, of which the two 3-core matmuls are 9.9 ms; item 6 first);
     - **MK7 output tail** (12.9 ms of untilize / pad / reshape in `out.pack`, plus the host conversion of item 7).
   - 2-4-op fusions to try first: the three-term conv triple + split + adds (item 2); conv + residual add + ReLU;
     the fp32 → bf16 typecast folded into the producing conv's output dtype; untilize + pad + reshape of each output
     into one pack kernel; the 4-op argmax chains.
   - A single whole-model megakernel is not realistic: 4,678 programs, 190 heterogeneous convs, 48.25 M parameters,
     77 MB BEV maps (SPEC §9).

## Raw logs, CSVs and reports

Paths are under `/home/ubuntu/experiments/tt-models/`. `L` = `logs/meteor/baseline/`.

| what | where |
|---|---|
| devrun jobs (`logs/devrun/history.log`, all rc 0, all with `METEOR_DEVICE_OK=1`, no tt-triage) | A 14:06:10-14:17:55 (`L/scripts/jobA_bench.sh`: stage bench, 4 inputs); B1 14:29:17-14:33:52 (`jobB_one.sh eth 2`); B2 14:36:03-14:51:10 (`jobB_one.sh worker 2`); C 14:51:59-15:13:28 (`jobC_profile.sh`: Tracy); D 15:26:43-15:27:46 (`jobD_visualizer.sh`); E 15:34:26-15:39:45 (`jobE_precision.sh`); F 15:41:19-15:43:36 (`jobF_rigswitch.sh`). Order: `L/scripts/run_rest.sh`, `run_e.sh`, `run_f.sh`; `L/run_all.rcs` |
| stage bench (ETH-1CQ) | `L/bench_eth-1cq.{json}`, `L/jobA_bench.log`; inputs `L/inputs/{pandaset_090_f40,nuscenes_0103_kf09}.json` (`L/scripts/make_manifests.py`) |
| dispatch matrix | `L/matrix_{eth-2cq,worker-2cq}.json`, `L/jobB_{eth-2cq,worker-2cq}.log` |
| rig switch | `L/rigswitch_eth-1cq.json`, `L/jobF_rigswitch.log` |
| precision cost | `L/precision_image-bf16.json`, `L/jobE_precision.log` |
| device profile | `generated/profiler/meteor_baseline/reports/2026_10_10_15_12_27/ops_perf_results_2026_10_10_15_12_27.csv` (+ `profile_log_device.csv.zst` (zstd-compressed 2026-10-10 to free disk; `zstd -d` restores the 6.0 GB CSV), `tracy_profile_log_host.tracy`; the ttnn-visualizer performance report); `L/profile_summary.{json,md}` (`code/scripts/profile_summary.py`); `L/profile_extra.json` (`L/scripts/profile_extra.py`: core histogram, per-module op codes, reshape shapes, typecast directions); `L/tt_perf_report_frame.{txt,csv}`, `L/tt_perf_report_summary_frame.{csv,png}`; `L/jobC_profile.log`, `L/profile_meta.json` |
| ttnn-visualizer graph report | `generated/ttnn_reports/meteor_baseline/frame/` (`db.sqlite`, `config.json`; `profile_ops.py --graph-report`, plain graph capture, 8.1 s, 52.7 MB), `L/jobD_visualizer.log`, `L/graph_report_meta.json` |
| host post-processing (CPU only, loaded host) | `L/host_post_profile.{json,log}` (`L/scripts/host_post_profile.py`) |
| host suite | `L/host_suite.log` (100 passed, 1 skipped) |
| gates of this code (`e86c13a`) | `logs/meteor/hangfix/suite_final{1,2,3}.log`, `values_final*`; the 2000-frame ETH stress `logs/meteor/hangfix/meteor_eth_final2000.*` |