# meteor-p150: verification of the first release, 2026-10-10

## Verdict

**PASS.** This is the light verification pass of the first publish (`research/PLAN.md` §6.3): the device suite
re-run with every frozen gate (read-only), the baseline numbers re-checked, the published configuration exercised end
to end (quickstart, host server + smoke test, a golden-free ETH stress), and every device-open path audited for ETH
dispatch. It was made by the agent that wrote the release docs, on the release tree (bundle commit `cc35027` plus the
release changes that this file is committed with). Against the baseline commit `e86c13a` (`OPT_BASELINE.md`,
committed as `cc35027`):

- **Accuracy:** the device suite passes (**16 passed** in one devrun job, 93 s). Every stored value
  (`pcc_<frame>.json` of the 4 frames, `e2e_device.json`, `variants_device.json`) is **identical** to the baseline
  code's three ETH suite runs (`logs/meteor/hangfix/values_final1/`), timings aside; the only new entry is the
  shipped synthetic sample's check (8 / 8 boxes, max |score difference| 0.0020, lane agreement 0.9987, path ADE
  0.018 m), gated with the existing thresholds. 38 public frames through the served graph agree with the CPU goldens
  (below).
- **Speed:** the device rows reproduce `OPT_BASELINE.md` to 0.15 ms (replay 556.92 vs 556.85 ms, back to back 556.89
  vs 556.75 ms); the end-to-end PandaSet call is 1.9 % slower (935.9 vs 918.2 ms, p50) on a busier shared host.
- **Audit:** ETH dispatch, 1 CQ and the 12×10 grid on every device-open path; no fallback in any run; both tt-metal
  patches present (`eth_patch: true`; the reshape-patch marker is asserted by the image build).
- **No numerics change of the device graph** since the verified port: `e86c13a..` touches no file under `tt/`, no
  precision knob and no vendored ttaw file (`vendor.py --check`: 0 differences, ttaw 0.23.2 @ common `60b6dd7`). The
  release changes are host-side and documented (OPT_REPORT.md "release" row). No gate threshold moved; the gate files
  are byte-identical.
- **No hang, no `.device.FAULT`, tt-triage never armed**: all 6 device jobs exited with rc 0 (every METEOR device job
  ran with the orchestrator's `METEOR_DEVICE_OK=1`).

The adversarial verification of the port is `VERIFY_PORT.md` (round 1, 2026-10-09: FAIL on two medium findings, V1
and V2, plus 7 low ones). Both mediums were fixed in PORT_LOG.md section 11 (E22, E23); "Review findings" below
lists where each finding stands. No second full verification round was run before this release.

## Setup

| item | value |
|---|---|
| Hardware | one Blackhole p150b, ETH dispatch, 1 CQ, 12×10 compute grid (120 cores), as printed by every run (`dispatch eth`, `fallback null`, `eth_patch true`); AICLK 1350 MHz (median of 163 samples during the bench, minimum 1343), board power median 69 W |
| tt-metal | `44d66500520` with `patches/tt-metal-eth-dispatch.patch` (sha256 `08d0ddf6…`) and `patches/tt-metal-reshape-rm-sys1419.patch` (sha256 `74878683…`), both byte-identical to the workspace copies the tree was built with |
| Weights | `AutowareFoundation/meteor` @ `01a5f6d71df5ecbbb5853ec600825481d57b9c6b` (tag `v1.0`) from the HF cache at the pinned revision (`HF_HUB_OFFLINE=1`, no token) |
| Verified code | `cc35027` + the release changes (ttaw 0.23.2 @ common `60b6dd7`, `vendor.py --check`: 0 differences) |
| Baseline | `e86c13a` (`OPT_BASELINE.md`) |
| Workloads | the shipped sample `samples/synthetic_8cam` (8 PNGs, 768×432, all cameras present, preset `synthetic_8cam`); the staged PandaSet 019 frame 40 sample (not shipped); the 4 golden frames of the gates (research goldens, local); PandaSet 019 / 090 and nuScenes scene-0103 frames of `research/meteor/public_data` (38 frames, local, never shipped) |
| Bench | `code/scripts/bench.py --inputs shipped,sample --iters 30 --b2b 20` (p50 / p99 of 30 per loop), one run |
| Host | AMD EPYC-Rome VM, 8 vCPUs, `OMP_NUM_THREADS=4`, shared with other agents' jobs (load average 6.0 → 8.7 during the bench) |
| Logs | `logs/meteor/release/` of the workspace (`job_{media,suite,quickstart,serve,bench,stress}.{sh,log}`, `run_all.rcs`); devrun windows 16:13:59, 16:30:01, 16:35:42, 16:38:44, 16:45:31, 16:50:22 UTC (`logs/devrun/history.log`), all rc 0 |

## Measured baseline vs re-check

| measurement | baseline `e86c13a` (`OPT_BASELINE.md`) | re-check (release tree) | change |
|---|---:|---:|---:|
| one blocking replay of `frame`, p50, PandaSet 019 | 556.85 ms (p99 559.34) | 556.92 ms (p99 557.5) | +0.07 ms |
| back-to-back replays, per frame | 556.75 ms | 556.89 ms | +0.14 ms |
| H2D (8 cameras + v0) | 66.5 ms | 66.6 ms | +0.1 ms |
| D2H (75 MB, 5 segments) | 17.2 ms (17-48 bimodal) | 46.4 ms | in the documented 17-48 ms range (host side) |
| host preprocess / camera transpose | 11.2 / 26.3 ms | 13.8 / 27.0 ms | host load |
| readback conversion (unpack + outputs) | 15.9 + 49.4 ms | 14.9 + 43.3 ms | host |
| host postprocess | 129.9 ms | 133.8 ms | host load |
| e2e `model()` PandaSet 019, p50 (p99) | 918.2 (1,071.8) ms | 935.9 (999.6) ms | +1.9 % |
| `from_pretrained` load, warm kernel cache | 49.7 s | 45.3 s (weights 0.3, build 2.3, warm-up 38.6 s) | −4.4 s |
| PandaSet 019 frame vs its stored CPU reference (bench check) | 8 / 8, recall / precision 1.0 / 1.0, max \|Δscore\| 0.0081, lane 0.99770, path 0.108 m | identical | 0 |

The device rows reproduce; the host rows move with the load of the shared host.

Not in the baseline, measured for the card on the release tree:

| measurement | value |
|---|---:|
| shipped sample through `model()` (bench, 30 iterations) | e2e **871.6 ms** (942.1); device 755.6, postprocess 95.2; replay 556.80 ms, back to back 556.65 ms |
| shipped sample vs its stored CPU reference (`test_e2e_device.py::test_shipped_sample_vs_stored_reference`, the bench check and the server smoke) | 8 / 8, recall 1.000, precision 1.000, max \|score difference\| 0.0020, centre error max 0.002 m, lane agreement 0.9987, path ADE / FDE 0.0177 / 0.0298 m, same plan mode |
| served `/predict`, host uvicorn, the smoke request (first request: includes the rig change of the new calibration) | `timing_ms.total` 1,226.0 ms (device 991.7), client round trip 1,250.7 ms |
| served `/predict`, three more requests of the same body (1.8 MB) | `timing_ms` decode 81.3-84.0 · preprocess 10.5-10.8 · device 719.5-722.4 · postprocess 86.1-87.1 · **total 904.0-908.7 ms**; client round trip 0.92-0.93 s |
| server boot to ready (warm kernel cache) | 44.6 s (`ready in 44.6 s`; warm-up 37.8 s) |
| served body vs the stored CPU reference (`server/smoke_test.py`, the container-smoke rule, `--expect VEHICLE:6,VRU:2`) | **PASS**: 8 / 8, recall 1.000, precision 1.000, dscore 0.0020; trajectory ADE 0.0177, FDE 0.0298; lane 0.9987; `/info` dispatch eth, grid 12x10, 1 CQ |
| `examples/quickstart.py` (from the repo root, default input) | rc 0: 8 boxes (6 VEHICLE, 2 VRU, scores 0.887-0.993), plan mode 0, traffic light red, 10 2D boxes; `quickstart.json` and `quickstart_bev.png` written |
| golden-free ETH stress, `stress_frames.py --mode rigs --rigs shipped --frames 200` (4 synthetic rigs, a rig change + table write + upload + replay + read every frame) | **200 frames clean**, 0 mismatches (4 stable per-rig digests); table write 164.6 ms, run 708.0 ms (max 784.8) |
| 38 public frames through the served graph (`code/scripts/run_public_frames.py`; every feed's sha256 equal to the CPU golden's) | 707-1,005 ms per frame (the first frame of each rig includes the table write) |

## Accuracy gate results

`test_pcc_device.py` + `test_e2e_device.py` + `test_variants_device.py` in one pytest, one devrun job,
`TTAW_GATES_READONLY=1`: **16 passed** (93 s). The gate files are unchanged (`git diff --stat` empty for
`*.gates.json`); the new shipped-sample test reuses the existing `det3d.*`, `ego.path_dev_m` and `lane.agreement`
gates and froze nothing.

| gate | threshold | value (identical to the baseline's ETH runs) |
|---|---:|---:|
| teacher-forced stage PCC, 26 gates (PandaSet 019 f40) | ≥ 0.99 | min 0.9999890 (`tf.risk`); `depth_mean` 0.99948 (reported) |
| teacher-forced lane / seg2d / depth argmax agreement | ≥ 0.99 | 0.9984 / 0.9965 / 0.9936 |
| chained frame, min output PCC: PandaSet 019 / 090, nuScenes 0103, valday #40 | ≥ 0.99 | 0.99989 / 0.99990 / 0.99991 / 0.99982 |
| chained lane / seg2d / depth agreement, worst frame | ≥ 0.99 | 0.9963 / 0.9965 / 0.9925 |
| replay == eager | bit-identical | yes (19 outputs) |
| det3d strict recall / precision: PandaSet sample, PandaSet 090, nuScenes, valday #40 | ≥ 0.95 / ≥ 0.95 | 1.0/1.0, 1.0/1.0, 1.0/1.0, 0.971/0.971 (34 / 35) |
| det3d near-tie extras (recall_nt ≥ recall, ambiguous_frac) | ≥ 0.95, ≤ 0.10 | pass on every frame |
| selected ego path deviation, worst frame | ≤ 0.3 m | 0.170 m (PandaSet 090) |
| lane agreement after the host decode, worst frame | ≥ 0.99 | 0.9966 |
| PandaSet sample `/predict` vs its stored reference body | ≥ 0.95 / ≥ 0.95 | 8 / 8, max \|Δscore\| 0.0081 |
| **shipped synthetic sample** `/predict` vs its stored reference body (new) | the same gates | 8 / 8, max \|Δscore\| 0.0020, path 0.030 m, lane 0.9987 |
| API == `/predict` (shipped sample) | identical bodies | yes |
| `input_norm=imagenet` + `depth_mean_bins=linear` variant, float outputs min PCC | ≥ 0.99 | 0.99988 |

**Public frames** (not gated by the device suite; `research/meteor/public_data/scripts/compare_tt.py` with the
PLAN thresholds, the TT outputs written by `run_public_frames.py`; `logs/meteor/release/public_tt/compare/`):

| sequence | frames | lane / seg2d / depth agreement (min) | dense-head PCC min (hm / reg / stationary / risk) | all 19 outputs, PCC min (one frame) | path dev max | 3D boxes: CPU matched / TT precision | same mode / TL |
|---|---:|---|---|---:|---:|---|---|
| PandaSet 019 | 13 | 0.99761 / 0.99637 / 0.99330 | 0.99995 / 0.99995 / 0.99999 / 0.99962 | 0.99948 (`depth_mean`) | 0.069 m | 99 / 103, 1.0 | 13 / 13 |
| PandaSet 090 | 13 | 0.99708 / 0.99684 / 0.99483 | 0.99996 / 0.99996 / 0.99999 / 0.99982 | 0.99983 | 0.110 m | 69 / 71, 1.0 | 13 / 13 |
| nuScenes scene-0103 | 12 | 0.99764 / 0.99624 / 0.99253 | 0.99997 / 0.99993 / 0.99998 / 0.99962 | 0.99902 | 0.081 m | 240 / 244, 1.0 | 12 / 12 |

Every frame passes the argmax, PCC and path gates. 8 of the 38 frames fall below the 95 % "golden boxes matched"
check of `compare_tt.py` because they have few boxes (5-14) and miss one or two: of the 10 missed CPU boxes, 9 score
within 0.004 of their class threshold on the CPU (0.3507-0.3538 vs 0.35; 0.1505-0.1513 vs 0.15), so a tiny score
difference drops them; the 10th is a pedestrian in a nuScenes crowd whose TT peak lies 2.1 m away. The TT never
produces a box the CPU does not (precision 1.0 on every frame). This is the documented near-threshold behaviour of a
bf16 / fp32 port (README Caveats), not a gate of the port: the frozen det3d gates run on the four golden frames above.

## Disclosed numerics changes

None on the device: the release does not touch `tt/`, the precision knobs or the vendored ttaw. The precision policy
is the port's (PLAN §9.3; PORT_LOG E20 / E22): fp32 weights and biases, HiFi4, fp32 dest and packer L1 accumulation;
the image branch and the 3D detection path with fp32 activations as three bf16-term convs; the BEV trunk in bf16.

Host-side changes of the release (no effect on any gate value, confirmed by the identical stored values):
- the shipped sample is now `samples/synthetic_8cam*` (generated, Apache-2.0); the PandaSet sample, its preset and
  its small goldens moved unchanged to the git-ignored `staging_samples_pandaset/` (sha256-checked), where
  `tests/paths.py` finds them (inside the workspace a missing staging dir fails those tests);
- `load_sample` resolves a preset from the `calib/` beside the manifest's `samples/` directory first (a staged sample
  carries its own preset) and returns it inline; `server/smoke_test.py` sends a staged preset inline;
- `opencv-python-headless>=4.8,<4.12` declared in `pyproject.toml` and `runtime.packages` (+ an `import cv2` verify
  line): the host post-processing needs OpenCV (PORT_LOG I3) and tt-metal's ttnn install does not bring it;
- `server/smoke_test.py`: the default sample and `DEFAULT_EXPECT = "VEHICLE:6,VRU:2"` (it was the unfilled template
  string (the template placeholder `SMOKE_EXPECT` in double braces), which the expectation parser would have read as a required label: the container smoke
  would have failed);
- `examples/quickstart.py` called `model(path)` (a path is the point-cloud argument: an `InputError`) and `len(out)`;
  fixed to `model(**load_sample(path))`, and it now writes the `quickstart_bev.png` it promised;
- `server/client.py`: `--sample <manifest>` and `--ego-speed` (the vendored client cannot send the required
  `ego_speed`);
- `stress_frames.py --rigs shipped[:N]`: rigs from the shipped sample, no golden needed (for the container check).

## Review findings

- No test threshold was loosened, no gate file edited; the device tests compare with goldens written by the fp32
  CPU reference before any device code existed (VERIFY_PORT 3(e)); the shipped sample's reference was written by the
  same CPU reference from the PNG files exactly as the API reads them (`make_synthetic_sample.py finalise` asserts the
  PNGs read back bit-exactly).
- The shipped sample was optimised against the CPU reference (object pixels only) so that its published boxes sit
  far from the thresholds (scores 0.889-0.993 vs 0.35 / 0.15; the strongest other heatmap cell 0.179 vehicle /
  0.072 VRU). This makes the container smoke stable; it does not make the TT output agree better (the optimisation
  never saw a TT output).
- Timing loops measure what they claim (`bench.py` synchronises each stage; the b2b row replays 20 frames back to
  back); every bench run also checks its own output against the stored reference.
- Nothing hung; the stress jobs ran contained (devrun timeout, no triage, no operation timeout).
- VERIFY_PORT round 1 findings:
  - V1 (det3d gate redefined after a failure): **fixed** by E22 (three-term det path): the strict PLAN gate is gated
    on every frame again, valday #40 0.971 / 0.971; re-confirmed here.
  - V2 (`input_norm=imagenet` never ran on the card): **fixed** by E23 (`test_variants_device.py`, also covering
    `depth_mean_bins=linear`, L1); re-confirmed here.
  - L1 (`IMAGE_PRECISION=bf16` exposed): the knob's documentation states it is below the depth gate (A/B only).
  - L2 (`depth_mean` max abs error from bin flips) and L3 (centre errors up to 0.4 m on public frames): reported
    metrics, unchanged.
  - L4 (fp32 D2H, host transpose): on the optimization backlog (OPT_REPORT items 3 and 7).
  - L5 (the API falls back to WORKER if the ETH open fails): unchanged by design (shared ttaw); the container smoke
    fails unless `/info` reports eth / 12x10, and the tests open with `allow_fallback=False`.
  - L6 (`_tables_key` set before the upload succeeded): fixed in the hang round (PORT_LOG 12.1 A3: the key is set
    after the last chunk lands). `tt/terms.py` stays unwired (documented). I3 (cv2 in the image): fixed in this
    release (declared + verify line). I4 (docs pointing at autoware_universe): fixed in this release (README,
    SERVING, card, pyproject point at tier4/METEOR).
  - L7: cosmetic.

## p150 ETH-dispatch compliance

| path | dispatch / CQs / grid |
|---|---|
| Python API `METEOR.from_pretrained()` | `DEVICE_DEFAULTS` (`device.py`): no dispatch override, so the vendored `ttaw.device` default ETH; 1 CQ; `l1_small_size` 32 KiB, `trace_region_size` 256 MiB. Every run of this pass printed `dispatch eth`, `grid 12x10`, `cores 120`, `fallback null`, `eth_patch true` |
| HTTP server (`METEOR_DISPATCH` default) | the same defaults; the host server run reported `/info` device `eth`, `12x10`, 1 CQ, `fallback null` |
| `tt-model.yaml` serve env (every serve profile) | one profile (`default`): `METEOR_DISPATCH: "eth"`, `METEOR_NUM_CQS: "1"`, `METEOR_VARIANT: "default"`; `check_bundle.py` refuses any other dispatch |
| image `verify:` | the ETH-patch marker in `topology.cpp`, `ttaw.device.eth_dispatch_patch_present()`, the reshape-patch marker (`dual_kernel_min_dram_dest_page_bytes`) in `reshape_rm_program_factory.cpp`, `import cv2`, the shipped sample and preset present |
| container smoke per serve profile (`/info` asserted: eth, 12x10, pins) | `server/smoke_test.py` hard-codes `EXPECT_DISPATCH, EXPECT_GRID = "eth", "12x10"` and checks the staged package's pins; run at packaging |
| gate tests (`code/conftest.py`) | `open_device(allow_fallback=False)`: a failing ETH open is an error, never a silent WORKER run |
| bench / profile / stress / public-frame scripts | `bench.py` and `profile_ops.py`: `--dispatch` default None = the API default (eth); `stress_frames.py`: `--dispatch eth` default with `allow_fallback=False`; `run_public_frames.py`: the API default, reports the device (eth, 12x10); the repro scripts default to eth |
| no hard-coded grid | `grep` for `CoreGrid`, `CoreRange`, `grid_x`, `core_grid` in `tt/` finds nothing (VERIFY_PORT 3(b)); WORKER 11×10 builds its own program set (OPT_BASELINE "Grid usage") |
| vendored `ttaw` | 0.23.2 @ common `60b6dd7` (`VENDORED.json`, `source_dirty: false`); `vendor.py --check`: 0 differences; `check_bundle.py` drift check: clean |