VERIFICATION_2026-10-10.md

meteor-p150: verification of the first release, 2026-10-10

Verdict

PASS. This is the light verification pass of the first publish (research/PLAN.md §6.3): the device suite re-run with every frozen gate (read-only), the baseline numbers re-checked, the published configuration exercised end to end (quickstart, host server + smoke test, a golden-free ETH stress), and every device-open path audited for ETH dispatch. It was made by the agent that wrote the release docs, on the release tree (bundle commit cc35027 plus the release changes that this file is committed with). Against the baseline commit e86c13a (OPT_BASELINE.md, committed as cc35027):

• Accuracy: the device suite passes (16 passed in one devrun job, 93 s). Every stored value (pcc_<frame>.json of the 4 frames, e2e_device.json, variants_device.json) is identical to the baseline code's three ETH suite runs (logs/meteor/hangfix/values_final1/), timings aside; the only new entry is the shipped synthetic sample's check (8 / 8 boxes, max |score difference| 0.0020, lane agreement 0.9987, path ADE 0.018 m), gated with the existing thresholds. 38 public frames through the served graph agree with the CPU goldens (below).
• Speed: the device rows reproduce OPT_BASELINE.md to 0.15 ms (replay 556.92 vs 556.85 ms, back to back 556.89 vs 556.75 ms); the end-to-end PandaSet call is 1.9 % slower (935.9 vs 918.2 ms, p50) on a busier shared host.
• Audit: ETH dispatch, 1 CQ and the 12×10 grid on every device-open path; no fallback in any run; both tt-metal patches present (eth_patch: true; the reshape-patch marker is asserted by the image build).
• No numerics change of the device graph since the verified port: e86c13a.. touches no file under tt/, no precision knob and no vendored ttaw file (vendor.py --check: 0 differences, ttaw 0.23.2 @ common 60b6dd7). The release changes are host-side and documented (OPT_REPORT.md "release" row). No gate threshold moved; the gate files are byte-identical.
• No hang, no .device.FAULT, tt-triage never armed: all 6 device jobs exited with rc 0 (every METEOR device job ran with the orchestrator's METEOR_DEVICE_OK=1).

The adversarial verification of the port is VERIFY_PORT.md (round 1, 2026-10-09: FAIL on two medium findings, V1 and V2, plus 7 low ones). Both mediums were fixed in PORT_LOG.md section 11 (E22, E23); "Review findings" below lists where each finding stands. No second full verification round was run before this release.

Setup

item	value
Hardware	one Blackhole p150b, ETH dispatch, 1 CQ, 12×10 compute grid (120 cores), as printed by every run (dispatch eth, fallback null, eth_patch true); AICLK 1350 MHz (median of 163 samples during the bench, minimum 1343), board power median 69 W
tt-metal	44d66500520 with patches/tt-metal-eth-dispatch.patch (sha256 08d0ddf6…) and patches/tt-metal-reshape-rm-sys1419.patch (sha256 74878683…), both byte-identical to the workspace copies the tree was built with
Weights	AutowareFoundation/meteor @ 01a5f6d71df5ecbbb5853ec600825481d57b9c6b (tag v1.0) from the HF cache at the pinned revision (HF_HUB_OFFLINE=1, no token)
Verified code	cc35027 + the release changes (ttaw 0.23.2 @ common 60b6dd7, vendor.py --check: 0 differences)
Baseline	e86c13a (OPT_BASELINE.md)
Workloads	the shipped sample samples/synthetic_8cam (8 PNGs, 768×432, all cameras present, preset synthetic_8cam); the staged PandaSet 019 frame 40 sample (not shipped); the 4 golden frames of the gates (research goldens, local); PandaSet 019 / 090 and nuScenes scene-0103 frames of research/meteor/public_data (38 frames, local, never shipped)
Bench	code/scripts/bench.py --inputs shipped,sample --iters 30 --b2b 20 (p50 / p99 of 30 per loop), one run
Host	AMD EPYC-Rome VM, 8 vCPUs, OMP_NUM_THREADS=4, shared with other agents' jobs (load average 6.0 → 8.7 during the bench)
Logs	logs/meteor/release/ of the workspace (job_{media,suite,quickstart,serve,bench,stress}.{sh,log}, run_all.rcs); devrun windows 16:13:59, 16:30:01, 16:35:42, 16:38:44, 16:45:31, 16:50:22 UTC (logs/devrun/history.log), all rc 0

Measured baseline vs re-check

measurement	baseline e86c13a (OPT_BASELINE.md)	re-check (release tree)	change
one blocking replay of frame, p50, PandaSet 019	556.85 ms (p99 559.34)	556.92 ms (p99 557.5)	+0.07 ms
back-to-back replays, per frame	556.75 ms	556.89 ms	+0.14 ms
H2D (8 cameras + v0)	66.5 ms	66.6 ms	+0.1 ms
D2H (75 MB, 5 segments)	17.2 ms (17-48 bimodal)	46.4 ms	in the documented 17-48 ms range (host side)
host preprocess / camera transpose	11.2 / 26.3 ms	13.8 / 27.0 ms	host load
readback conversion (unpack + outputs)	15.9 + 49.4 ms	14.9 + 43.3 ms	host
host postprocess	129.9 ms	133.8 ms	host load
e2e model() PandaSet 019, p50 (p99)	918.2 (1,071.8) ms	935.9 (999.6) ms	+1.9 %
from_pretrained load, warm kernel cache	49.7 s	45.3 s (weights 0.3, build 2.3, warm-up 38.6 s)	−4.4 s
PandaSet 019 frame vs its stored CPU reference (bench check)	8 / 8, recall / precision 1.0 / 1.0, max |Δscore| 0.0081, lane 0.99770, path 0.108 m	identical	0

The device rows reproduce; the host rows move with the load of the shared host.

Not in the baseline, measured for the card on the release tree:

measurement	value
shipped sample through model() (bench, 30 iterations)	e2e 871.6 ms (942.1); device 755.6, postprocess 95.2; replay 556.80 ms, back to back 556.65 ms
shipped sample vs its stored CPU reference (test_e2e_device.py::test_shipped_sample_vs_stored_reference, the bench check and the server smoke)	8 / 8, recall 1.000, precision 1.000, max |score difference| 0.0020, centre error max 0.002 m, lane agreement 0.9987, path ADE / FDE 0.0177 / 0.0298 m, same plan mode
served /predict, host uvicorn, the smoke request (first request: includes the rig change of the new calibration)	timing_ms.total 1,226.0 ms (device 991.7), client round trip 1,250.7 ms
served /predict, three more requests of the same body (1.8 MB)	timing_ms decode 81.3-84.0 · preprocess 10.5-10.8 · device 719.5-722.4 · postprocess 86.1-87.1 · total 904.0-908.7 ms; client round trip 0.92-0.93 s
server boot to ready (warm kernel cache)	44.6 s (ready in 44.6 s; warm-up 37.8 s)
served body vs the stored CPU reference (server/smoke_test.py, the container-smoke rule, --expect VEHICLE:6,VRU:2)	PASS: 8 / 8, recall 1.000, precision 1.000, dscore 0.0020; trajectory ADE 0.0177, FDE 0.0298; lane 0.9987; /info dispatch eth, grid 12x10, 1 CQ
examples/quickstart.py (from the repo root, default input)	rc 0: 8 boxes (6 VEHICLE, 2 VRU, scores 0.887-0.993), plan mode 0, traffic light red, 10 2D boxes; quickstart.json and quickstart_bev.png written
golden-free ETH stress, stress_frames.py --mode rigs --rigs shipped --frames 200 (4 synthetic rigs, a rig change + table write + upload + replay + read every frame)	200 frames clean, 0 mismatches (4 stable per-rig digests); table write 164.6 ms, run 708.0 ms (max 784.8)
38 public frames through the served graph (code/scripts/run_public_frames.py; every feed's sha256 equal to the CPU golden's)	707-1,005 ms per frame (the first frame of each rig includes the table write)

Accuracy gate results

test_pcc_device.py + test_e2e_device.py + test_variants_device.py in one pytest, one devrun job, TTAW_GATES_READONLY=1: 16 passed (93 s). The gate files are unchanged (git diff --stat empty for *.gates.json); the new shipped-sample test reuses the existing det3d.*, ego.path_dev_m and lane.agreement gates and froze nothing.

gate	threshold	value (identical to the baseline's ETH runs)
teacher-forced stage PCC, 26 gates (PandaSet 019 f40)	≥ 0.99	min 0.9999890 (tf.risk); depth_mean 0.99948 (reported)
teacher-forced lane / seg2d / depth argmax agreement	≥ 0.99	0.9984 / 0.9965 / 0.9936
chained frame, min output PCC: PandaSet 019 / 090, nuScenes 0103, valday #40	≥ 0.99	0.99989 / 0.99990 / 0.99991 / 0.99982
chained lane / seg2d / depth agreement, worst frame	≥ 0.99	0.9963 / 0.9965 / 0.9925
replay == eager	bit-identical	yes (19 outputs)
det3d strict recall / precision: PandaSet sample, PandaSet 090, nuScenes, valday #40	≥ 0.95 / ≥ 0.95	1.0/1.0, 1.0/1.0, 1.0/1.0, 0.971/0.971 (34 / 35)
det3d near-tie extras (recall_nt ≥ recall, ambiguous_frac)	≥ 0.95, ≤ 0.10	pass on every frame
selected ego path deviation, worst frame	≤ 0.3 m	0.170 m (PandaSet 090)
lane agreement after the host decode, worst frame	≥ 0.99	0.9966
PandaSet sample /predict vs its stored reference body	≥ 0.95 / ≥ 0.95	8 / 8, max |Δscore| 0.0081
shipped synthetic sample /predict vs its stored reference body (new)	the same gates	8 / 8, max |Δscore| 0.0020, path 0.030 m, lane 0.9987
API == /predict (shipped sample)	identical bodies	yes
input_norm=imagenet + depth_mean_bins=linear variant, float outputs min PCC	≥ 0.99	0.99988

Public frames (not gated by the device suite; research/meteor/public_data/scripts/compare_tt.py with the PLAN thresholds, the TT outputs written by run_public_frames.py; logs/meteor/release/public_tt/compare/):

sequence	frames	lane / seg2d / depth agreement (min)	dense-head PCC min (hm / reg / stationary / risk)	all 19 outputs, PCC min (one frame)	path dev max	3D boxes: CPU matched / TT precision	same mode / TL
PandaSet 019	13	0.99761 / 0.99637 / 0.99330	0.99995 / 0.99995 / 0.99999 / 0.99962	0.99948 (depth_mean)	0.069 m	99 / 103, 1.0	13 / 13
PandaSet 090	13	0.99708 / 0.99684 / 0.99483	0.99996 / 0.99996 / 0.99999 / 0.99982	0.99983	0.110 m	69 / 71, 1.0	13 / 13
nuScenes scene-0103	12	0.99764 / 0.99624 / 0.99253	0.99997 / 0.99993 / 0.99998 / 0.99962	0.99902	0.081 m	240 / 244, 1.0	12 / 12

Every frame passes the argmax, PCC and path gates. 8 of the 38 frames fall below the 95 % "golden boxes matched" check of compare_tt.py because they have few boxes (5-14) and miss one or two: of the 10 missed CPU boxes, 9 score within 0.004 of their class threshold on the CPU (0.3507-0.3538 vs 0.35; 0.1505-0.1513 vs 0.15), so a tiny score difference drops them; the 10th is a pedestrian in a nuScenes crowd whose TT peak lies 2.1 m away. The TT never produces a box the CPU does not (precision 1.0 on every frame). This is the documented near-threshold behaviour of a bf16 / fp32 port (README Caveats), not a gate of the port: the frozen det3d gates run on the four golden frames above.

Disclosed numerics changes

None on the device: the release does not touch tt/, the precision knobs or the vendored ttaw. The precision policy is the port's (PLAN §9.3; PORT_LOG E20 / E22): fp32 weights and biases, HiFi4, fp32 dest and packer L1 accumulation; the image branch and the 3D detection path with fp32 activations as three bf16-term convs; the BEV trunk in bf16.

Host-side changes of the release (no effect on any gate value, confirmed by the identical stored values):

• the shipped sample is now samples/synthetic_8cam* (generated, Apache-2.0); the PandaSet sample, its preset and its small goldens moved unchanged to the git-ignored staging_samples_pandaset/ (sha256-checked), where tests/paths.py finds them (inside the workspace a missing staging dir fails those tests);
• load_sample resolves a preset from the calib/ beside the manifest's samples/ directory first (a staged sample carries its own preset) and returns it inline; server/smoke_test.py sends a staged preset inline;
• opencv-python-headless>=4.8,<4.12 declared in pyproject.toml and runtime.packages (+ an import cv2 verify line): the host post-processing needs OpenCV (PORT_LOG I3) and tt-metal's ttnn install does not bring it;
• server/smoke_test.py: the default sample and DEFAULT_EXPECT = "VEHICLE:6,VRU:2" (it was the unfilled template string (the template placeholder SMOKE_EXPECT in double braces), which the expectation parser would have read as a required label: the container smoke would have failed);
• examples/quickstart.py called model(path) (a path is the point-cloud argument: an InputError) and len(out); fixed to model(**load_sample(path)), and it now writes the quickstart_bev.png it promised;
• server/client.py: --sample <manifest> and --ego-speed (the vendored client cannot send the required ego_speed);
• stress_frames.py --rigs shipped[:N]: rigs from the shipped sample, no golden needed (for the container check).

Review findings

• No test threshold was loosened, no gate file edited; the device tests compare with goldens written by the fp32 CPU reference before any device code existed (VERIFY_PORT 3(e)); the shipped sample's reference was written by the same CPU reference from the PNG files exactly as the API reads them (make_synthetic_sample.py finalise asserts the PNGs read back bit-exactly).
• The shipped sample was optimised against the CPU reference (object pixels only) so that its published boxes sit far from the thresholds (scores 0.889-0.993 vs 0.35 / 0.15; the strongest other heatmap cell 0.179 vehicle / 0.072 VRU). This makes the container smoke stable; it does not make the TT output agree better (the optimisation never saw a TT output).
• Timing loops measure what they claim (bench.py synchronises each stage; the b2b row replays 20 frames back to back); every bench run also checks its own output against the stored reference.
• Nothing hung; the stress jobs ran contained (devrun timeout, no triage, no operation timeout).
• VERIFY_PORT round 1 findings:
  • V1 (det3d gate redefined after a failure): fixed by E22 (three-term det path): the strict PLAN gate is gated on every frame again, valday #40 0.971 / 0.971; re-confirmed here.
  • V2 (input_norm=imagenet never ran on the card): fixed by E23 (test_variants_device.py, also covering depth_mean_bins=linear, L1); re-confirmed here.
  • L1 (IMAGE_PRECISION=bf16 exposed): the knob's documentation states it is below the depth gate (A/B only).
  • L2 (depth_mean max abs error from bin flips) and L3 (centre errors up to 0.4 m on public frames): reported metrics, unchanged.
  • L4 (fp32 D2H, host transpose): on the optimization backlog (OPT_REPORT items 3 and 7).
  • L5 (the API falls back to WORKER if the ETH open fails): unchanged by design (shared ttaw); the container smoke fails unless /info reports eth / 12x10, and the tests open with allow_fallback=False.
  • L6 (_tables_key set before the upload succeeded): fixed in the hang round (PORT_LOG 12.1 A3: the key is set after the last chunk lands). tt/terms.py stays unwired (documented). I3 (cv2 in the image): fixed in this release (declared + verify line). I4 (docs pointing at autoware_universe): fixed in this release (README, SERVING, card, pyproject point at tier4/METEOR).
  • L7: cosmetic.

p150 ETH-dispatch compliance

path	dispatch / CQs / grid
Python API METEOR.from_pretrained()	DEVICE_DEFAULTS (device.py): no dispatch override, so the vendored ttaw.device default ETH; 1 CQ; l1_small_size 32 KiB, trace_region_size 256 MiB. Every run of this pass printed dispatch eth, grid 12x10, cores 120, fallback null, eth_patch true
HTTP server (METEOR_DISPATCH default)	the same defaults; the host server run reported /info device eth, 12x10, 1 CQ, fallback null
tt-model.yaml serve env (every serve profile)	one profile (default): METEOR_DISPATCH: "eth", METEOR_NUM_CQS: "1", METEOR_VARIANT: "default"; check_bundle.py refuses any other dispatch
image verify:	the ETH-patch marker in topology.cpp, ttaw.device.eth_dispatch_patch_present(), the reshape-patch marker (dual_kernel_min_dram_dest_page_bytes) in reshape_rm_program_factory.cpp, import cv2, the shipped sample and preset present
container smoke per serve profile (/info asserted: eth, 12x10, pins)	server/smoke_test.py hard-codes EXPECT_DISPATCH, EXPECT_GRID = "eth", "12x10" and checks the staged package's pins; run at packaging
gate tests (code/conftest.py)	open_device(allow_fallback=False): a failing ETH open is an error, never a silent WORKER run
bench / profile / stress / public-frame scripts	bench.py and profile_ops.py: --dispatch default None = the API default (eth); stress_frames.py: --dispatch eth default with allow_fallback=False; run_public_frames.py: the API default, reports the device (eth, 12x10); the repro scripts default to eth
no hard-coded grid	grep for CoreGrid, CoreRange, grid_x, core_grid in tt/ finds nothing (VERIFY_PORT 3(b)); WORKER 11×10 builds its own program set (OPT_BASELINE "Grid usage")
vendored ttaw	0.23.2 @ common 60b6dd7 (VENDORED.json, source_dirty: false); vendor.py --check: 0 differences; check_bundle.py drift check: clean