# meteor-p150 optimization report (p150, ETH dispatch, 12×10 grid)

**Status: baseline port; optimization pending.** This is the first public release: the functional port, measured in
[`OPT_BASELINE.md`](OPT_BASELINE.md) and published with the precision policy of the first release (fp32 weights and
biases, HiFi4, fp32 and packer L1 accumulation; the image branch and the 3D detection path with fp32 activations,
every conv as three bf16 terms; the BEV trunk with bf16 activations). No optimization round has run yet. The whole
network runs as ONE metal trace of **4,678 programs**, which replays in **556.85 ms** (back to back 556.75 ms,
1.80 frames/s device-bound). A synchronous `model()` call takes 876-918 ms (p50, by input): around the replay come
the host decode (124-131 ms), the conversion of the 75 MB readback (61-65 ms), the image upload (66 ms) and the host
camera transpose (22-26 ms). The trace is kernel-bound (2.74 ms of op-to-op gaps), and 57 % of the kernel time moves
or reformats data rather than computing. The ranked plan below starts with the TILE reshape copies and the three-term
convs. Every round will be added to this file, newest first, with its commits, its numbers and its accuracy gates.

All numbers: `code/scripts/bench.py` (warm, batch 1, p50 of 60 iterations unless marked), ETH dispatch, 1 CQ,
12×10 grid, AICLK 1350 MHz (minimum 1343), tt-metal `44d66500520` + `patches/tt-metal-eth-dispatch.patch` +
`patches/tt-metal-reshape-rm-sys1419.patch`, measured 2026-10-10 on the baseline commit `e86c13a` (ttaw 0.23.2;
OPT_BASELINE.md committed as `cc35027`). Device profile: `code/scripts/profile_ops.py` under the Tracy device profiler
(two traced replays). Accuracy gates: `OPT_BASELINE.md` "How to run".

## Summary

| | baseline `e86c13a` | **this release** |
|---|---|---|
| device, one blocking replay of `frame` | 556.85 ms (p99 559.34) | same graph (no device code changed); re-measured **556.92 ms** (p99 557.5) on the PandaSet frame, 556.80 ms on the shipped sample |
| back-to-back replays | 556.75 ms per frame (1.80 frames/s) | re-measured **556.89 / 556.65 ms** |
| e2e `model()` p50 (p99): PandaSet 019 sample / synthetic noise / PandaSet 090 / nuScenes 0103 | 918.2 (1,071.8) / 875.7 (901.8) / 902.8 (927.8) / 914.4 (931.1) ms | PandaSet 019 935.9 (999.6) ms on a busier host (30 iterations); the new shipped synthetic sample 871.6 (942.1) ms; served `/predict` `timing_ms.total` 904-909 ms |
| host preprocess / H2D / D2H / readback conversion / postprocess (PandaSet 019) | 11.2 / 66.5 / 17.2 (17-48, bimodal) / 65.3 / 129.9 ms | unchanged |
| rig change (lift tables of a new calibration) | 182.3 ms table write; call 1,128.5 ms p50 | unchanged |
| device programs per frame | 4,678 (1,131 program-cache entries) | same trace |
| kernel sum / span (profile) | 555.28 / 558.02 ms; 2.74 ms of op-to-op gaps | |
| image branch / lift / BEV trunk / heads + planner / output tail (kernel ms) | 268.27 / 56.26 / 147.48 / 70.34 / 12.93 | |
| accuracy gates | every PLAN 2.13 gate green, frozen values (OPT_BASELINE.md "Accuracy") | the same frozen gates (values in VERIFICATION_2026-10-10.md) + the container smoke's check on the new shipped synthetic sample |

## Steps (chronological; each row is one commit)

| step | commit | what | b2b frame | e2e (PandaSet 019 p50) | accuracy | revert switch |
|---|---|---|---|---|---|---|
| baseline | `e86c13a` | the verified functional port (VERIFY_PORT round 1 fixed; ETH default with the SYS-1419 reshape fix, ttaw 0.23.2) | 556.75 ms | 918.2 ms | every gate green, frozen values; 2000-frame ETH rig-switch stress clean | |
| release | the release commit (`git log`: "Release docs + demo media") | release changes, none on the device graph: the synthetic shipped sample (`samples/synthetic_8cam*`); the PandaSet sample moved unchanged to the git-ignored staging dir; `opencv-python-headless` declared for the image (host post-processing, PORT_LOG I3); the smoke test's sample expectation filled; `server/client.py --sample / --ego-speed`; `examples/quickstart.py` fixed (it called `model(path)`); `stress_frames.py --rigs shipped` (golden-free, for the container) | 556.89 ms | 935.9 ms (busier host) | frozen gates unchanged: 16 device tests passed, every stored value identical to the baseline's ETH runs; the new shipped-sample check 8 / 8 | |

## Round 1

Not started. The first items are fixed by the baseline profile (below and OPT_BASELINE.md).

## Known hangs (all rounds)

All hangs below happened before this baseline, during the port (PORT_LOG.md section 12;
`logs/meteor/hangfix/ETH_DISPATCH_HANG_ISSUE.md` of the workspace). None happened at or after `e86c13a`.

| when (UTC) | command | cause | status |
|---|---|---|---|
| 10-08 / 10-09 (4 jobs) | METEOR device suites and benches under ETH dispatch | a non-deterministic hang of the trace replay; tt-triage launched on the hung chip then hard-reset the server 4 times | triage is now refused by the workspace's device lock; root cause below |
| 10-10 04:17-05:38 (3 hangs) | `stress_frames.py --mode rigs` under ETH 1 CQ (frames 263, 248) and ETH 2 CQs (frame 43); WORKER 1,600 frames clean | ETH dispatch implicated; contained (no triage, no reset; the host stayed up) | bisected |
| 10-10 07:15-09:28 (4 hangs + repros) | segmented-trace bisect: lift trace -> `lift_rest` -> `Resize2d` (400×250 -> 800×500) -> its tail -> one op | **one row-major `ttnn.reshape` [800, 48000] -> [400000, 96] (dual-kernel, 192 / 384 B DRAM pages from both NOCs) hangs under ETH dispatch: Blackhole DRAM-arbiter issue SYS-1419**; reproduced standalone in plain ttnn (`code/scripts/repro_resize2d_eth.py`) | **fixed**: `patches/tt-metal-reshape-rm-sys1419.patch` (single-kernel path for small DRAM destination pages; 20,000 replays clean at 192 / 384 B) applied to the image's tt-metal; ttaw 0.23.2 keeps a TILE tail when the patch is absent |
| 10-10 11:4x-13:51 | after the fix: device suite ×3, a 2000-frame ETH rig-switch stress at `18e0fbd` and another at `e86c13a`, and the baseline's 7 device jobs | none | ETH is the default again |

## Rejected / not kept

From the port and the baseline (PORT_LOG.md E20 / E22, OPT_BASELINE.md "precision cost"), kept here so the
optimization rounds do not repeat them blindly:

- **bf16 activations in the image branch** (`METEOR_IMAGE_PRECISION=bf16`): −165.9 ms per frame (−29.8 %), the
  largest single saving measured, but it fails the depth argmax gate (agreement 0.9797 < 0.99): the 64 depth bins are
  near-tied (13 % of the pixels have a top-2 logit margin < 0.02). The shipped sample's loose smoke comparison still
  passes with it, which shows how much stricter the frozen gates are.
- **ttnn's own fp32-activation conv path**: worse than bf16 (depth agreement 0.9617): it is not TF32 quality.
- **Two-term activations with fp32 weights**: emulated 0.9940 on the depth argmax gate as a whole (per-layer two
  terms are untested).
- **A bf16 det stem / box refiner** (`det_precision="bf16"`): −41 ms, but the strict det3d recall on the in-domain
  METEOR frame falls to 0.943 < 0.95 (PORT_LOG E22).
- **The near-tie aware det3d gate** (E21): withdrawn; the strict PLAN gate is kept on every frame.
- **WORKER dispatch as the default**: 22.1 ms slower per frame (578.9 ms, 11×10) and a 13-minute JIT compile; it was
  the default only while the ETH hang was open.
- **A second command queue**: ETH-2CQ replays 4.6 ms slower and a synchronous request gains nothing (D14).

## Profile at the end (trace replay)

The baseline profile is the current one (OPT_BASELINE.md "Device profile"): ReshapeView 128.5 ms (23.1 %),
BinaryNg 85.7, PaddedSlice / SliceWrite / Slice 72.6, Conv2d 65.2, Matmul 54.3, Typecast 50.9, Tilize / Untilize / Pad
36.3, GridSample 28.2 ms.

## Remaining backlog (gains on the device frame, 556.8 ms, unless marked e2e)

The ranked opportunities of OPT_BASELINE.md; the estimates overlap and do not add up. Each item keeps the frozen
gates and the precision policy (PLAN §9.3) and lands as one commit with one `METEOR_*` A/B knob.

1. **TILE reshape copies** (−90..−120 ms): 435 `ReshapeView` copies (128.5 ms) between `[1, 1, N·H·W, C]` and
   `[1, H, W, C]` at map widths that are not multiples of 32 (500, 250, 48, ...). One layout per stage, a flattened
   input path for `Resize2d`, ROW_MAJOR views where page-aligned (safe again under ETH with the SYS-1419 patch), or
   padding the BEV width to 512 / 256. The same C17 / C24 item as CenterPoint and BEVFormer (shared ttaw fix).
2. **Three-term convs as one fused op** (−90..−150 ms, precision kept): a conv kernel that splits hi / lo in the
   reader and accumulates the three products in the fp32 dest removes 359 typecasts (50.9 ms), most of the 440
   BinaryNg adds and two of three activation reads. Or a TF32+-quality fp32-activation conv (I7).
3. **Input path and stem** (−35..−45 ms; e2e −60 ms): the uint8 -> bf16 typecast over 3-byte pages (8.6 ms), the
   stem's 6 DRAM slices of the 3-channel input (13.9 ms), the exact fp32 max pool as a DRAM chain (25.3 ms; a
   generic_op pool about 1-3 ms, UNVERIFIED), and the H2D of 2.65 M pages of 3 bytes (66.4 ms; whole camera rows
   about 2-5 ms, UNVERIFIED).
4. **DRAM slicing and conv glue** (−25..−40 ms): 386 PaddedSlice + 386 SliceWrite (49.3 ms) and the halo / sharding
   glue (19.9 ms) of the DRAM-sliced convs.
5. **Lift as one gather kernel** (−30..−45 ms): two `grid_sample` calls of 14.1 ms writing 256 MB, then tilize,
   typecast, the product with the 205 MB fp32 table and the sums; a kernel per BEV block with L1-resident camera maps
   and per-calibration ELL weights writes only `[400, 250, 96]`.
6. **Few-core and fp32 matmuls** (−12..−20 ms): `head.ego_attn` (5.10 ms) and `head.risk_sample` (4.81 ms) run on 3
   cores; the W-axis `Resize2d` matmuls at 250 -> 500 on 16-24 cores (2.9 ms each).
7. **Host side** (e2e −150..−220 ms): the host decode (124-131 ms: the seg fusion and the 2D peak loops), the readback
   conversion (61-65 ms) and D2H of 75 MB fp32 (bf16 / uint8 outputs in final layouts), the NCHW -> NHWC transpose
   (22-26 ms).
8. **Throughput by pipelining** (frames/s, latency unchanged): 2 CQs with double-buffered I/O, bounded by the
   device's 1.8 frames/s.
9. **Megakernels and fusions** (D19): MK4 band-streamed BEV trunk (197 ms of BEV + refiners, 91 ms of it reshapes)
   first, then MK1 / MK2 image encoder + heads (after item 2), MK3 lift, MK0 input, MK6 planner, MK7 output tail.
   Dispatch is not a sink (0.5 % of the frame), so a megakernel pays only through the DRAM round trips and layout
   copies it removes.