VERIFICATION_2026-10-09.md

frnet-p150: independent verification, 2026-10-09

Verdict

PASS. This is the light verification pass of the first publish (research/PLAN.md §6.3): the frozen gates re-run, the baseline numbers re-checked, and every device-open path audited for ETH dispatch. It was made by the agent that wrote the release docs, which is neither the porter, nor the adversarial verifier of VERIFY_PORT.md, nor the baseline agent, on the release-code commit 04c39b5 (the model code of the baseline 0dd52b2; the shipped sample became a synthetic scan per serve profile, the PandaSet sample is staged and not shipped, test hardening; the docs commit that follows changes make_demo.py's render captions and the docs only). The device runs ran on 2026-10-09 15:53-16:10 UTC in one bin/devrun window (a first queueing attempt died while waiting for the lock, when the workspace's bin/devrun was edited under it; it had not touched the device). Against the baseline commit 0dd52b2:

• Frozen gates: the device suite (test_pcc_device.py + test_e2e_device.py, one pytest) passed 14 / 14 tests. Every existing gate passes with values bit-identical to the baseline run (the _m5 and _m5docs reports: 2,144 leaves compared, 0 differ; timings excluded), including the binding frame: OT128 min agreement 98.907 % (nuScenes 0916 kf40) against the 98.5 % gate. The two new tests (test_shipped_sample, and test_public_frames now requiring all 17 / 5 frames) passed; the four new shipped-sample gates were frozen at this first green run (label agreement OT128 0.99550, QT128 0.99623; filtered-cloud agreement 0.99772 / 0.99872; threshold 0.985).
• Baseline re-check: the device rows of OPT_BASELINE.md repeat to within 0.03 ms (OT128 210.02 / 209.95 ms trace / b2b on the synthetic sample vs 210.00 / 209.95 ms now; QT128 133.53 / 133.43 vs 133.50 / 133.43 ms). Host rows within the load noise (OT128 model() 292.6 -> 289.3 ms p50 on the synthetic sample, 355.7 -> 360.1 ms on PandaSet, at load average 4.1).
• Shipped samples: the shipped synthetic_<variant>.npz are byte-identical to the baseline's synthetic bench inputs (cmp), so the baseline's synthetic rows are the shipped sample's. p150 vs the stored CPU reference: OT128 99.550 % of 52,613 points, QT128 99.623 % of 69,553, the same through bench.py, the device test, the quickstart, make_demo.py and the served smoke test.
• Public data (113 frames per profile, production API): pooled class agreement OT128 PandaSet 99.66 % / nuScenes 99.19 %, QT128 99.29 % / 99.44 %; identical scored rows to the CPU reference on all 226 frame-profile pairs, no profile skip. 3 pairs fall below 98.5 % (not gated; see "Review findings"). Mapped-label mIoU on the p150 equals the CPU reference's within 0.002.
• ETH dispatch: every device-open path opens ETH / 1 CQ / 12×10 by default; every run of this pass printed dispatch eth, grid 12x10, cores 120, fallback null, eth_patch true (both serve profiles, API and server).
• No numerics change and no new bug: nothing in the model code was changed by this pass.

Setup

item	value
Hardware	one Blackhole p150b, ETH dispatch, 1 CQ, 12×10 compute grid (120 cores), AICLK 1350 MHz median (min 1343) during the benches
tt-metal	44d66500520 with patches/tt-metal-eth-dispatch.patch (sha256 08d0ddf6…, byte-identical to the workspace copy)
Weights	AutowareFoundation/lidar_frnet @ a53a1c11b0b28d31fd50a03abef5db0e830a5b3c (tag v2.0): the device suite and the demo run read the sha256-pinned workspace copy; the quickstart, the served runs and the load-time runs resolved the HF cache at the pinned revision offline (downloaded without a token for this pass; every file identical to the workspace copy, ONNX sha256 ot128 d29bf595…, qt128 647af927…)
Verified commit	04c39b5 (ttaw 0.20.0, common 89dec49, source_dirty: false)
Baseline commit	0dd52b2 (OPT_BASELINE.md)
Workload	the shipped samples code/tt_frnet/samples/synthetic_{ot128,qt128}.npz and the staged PandaSet 019 f40 (107,146 points, local), batch 1, warm; the 17 / 5 gate frames; 113 converted public frames per profile (local only)
Bench method	code/scripts/bench.py --iters 100 --warmup 10 --b2b-iters 50 --b2b-rounds 5, one run per serve profile (the baseline's settings), with each output checked against its stored CPU reference
Served method	uvicorn on the host with the serve pins of tt-model.yaml (the container's app; FRNET_VARIANT per profile), the bundle's smoke_test.py against it, then 5 warm-up + 50 requests of the shipped sample from a loopback client, the README's client.py + curl recipe, a knobs request and a request without calibration
Accuracy reference	the fp32 CPU reference (code/tt_frnet/reference/, ONNX Runtime on the deployed ONNX; VERIFY_PORT: within 1e-11 of an independent ORT run); for the 113-frame sets the research ORT runs (research/frnet/public_data/runs)
Host	AMD EPYC VM, 8 vCPUs, shared with other agents' jobs (load average 4.1-6.1 during the runs)
Logs	logs/frnet/m5_docs/ of the workspace (job scripts in scripts/)

Measured baseline vs re-check

All times in ms, p50 of 100 iterations (b2b: 5 rounds of 50 replays). Baseline: OPT_BASELINE.md (0dd52b2); re-check: code/scripts/bench.py with the baseline's settings on 04c39b5 (logs/frnet/m5_docs/bench_recheck_{ot128,qt128}.json).

row	baseline	re-check
OT128, shipped sample: trace, one replay / back-to-back	210.02 / 209.948	210.00 / 209.946
OT128, PandaSet 019 f40: trace / back-to-back	211.72 / 211.643	211.73 / 211.639
QT128, shipped sample: trace / back-to-back	133.53 / 133.434	133.50 / 133.429
QT128, PandaSet 019 f40: trace / back-to-back	134.91 / 134.866	134.92 / 134.856
OT128 H2D / D2H, shipped sample	5.42 / 7.65	5.42 / 7.56
OT128 host pre-processing, shipped sample / PandaSet	55.3 / 107.2	54.7 / 107.4
OT128 model() p50, shipped sample / PandaSet	292.6 / 355.7	289.3 / 360.1
QT128 model() p50, shipped sample / PandaSet	221.1 / 244.3	205.4 / 241.5
served /predict timing_ms.total, OT128 / QT128 (median of 50)	–	310.4 / 228.9
served client round trip, loopback	–	328.1 / 247.3
from_pretrained, empty JIT cache / warm cache, OT128	– / 6.5 s	156.6 / 7.5 s
from_pretrained, empty JIT cache / warm cache, QT128	– / 6.9 s	151.9 / 6.6 s

The served and cold-load rows are new for the card (the baseline measured neither). The first two calls after a load took 320.0 / 304.7 ms (OT128) and 241.1 / 214.9 ms (QT128): the trace is captured during the load. The cold JIT cache holds 682 MB (OT128) / 644 MB (QT128) after the load.

Accuracy gate results

logs/frnet/m5_docs/suite.log; reports logs/frnet/{pcc,pcc_small,e2e,latency}_{ot128,qt128}_m5docs.json.

check	gate	OT128	QT128
every gated module tap (PCC vs the fp32 reference, PandaSet sample, replay outputs)	≥ 0.999	min 0.999729 (point_fusion1)	min 0.999649 (attention4)
pred_probs PCC / raw-point agreement, PandaSet sample (staged)	≥ 0.998 / ≥ 98.5 %	0.999942 / 99.581 %	0.999844 / 99.295 %
public frames (17 / 5): min pred_probs PCC	≥ 0.998	0.999688	0.999805
public frames: min raw-point agreement	≥ 98.5 %	98.907 % (nuScenes 0916 kf40)	99.158 % (PandaSet 029 f40)
shipped sample vs its stored reference body: labels / filtered-cloud mask (new)	≥ 98.5 %	99.550 % / 99.772 %	99.623 % / 99.872 %
replay == eager	bit-exact	45 / 45 taps	45 / 45 taps
API == /predict handler	identical	yes	yes

Host suite (fake ttnn, gates read-only): 117 passed, 15 skipped (the 14 device tests and the onnxruntime-only test), logs/frnet/m5_docs/host_suite.log. Research venv (ONNX Runtime + torch): 85 passed, 2 skipped (by design), logs/frnet/m5_docs/research_venv.log. The counts differ from VERIFY_PORT's (112 / 13 host, 83 / 2 research venv) by the tests the release added or changed (release hygiene of the shipped samples, the top-level PYTHON.md copy, the shipped-sample device test).

Public-frame agreement beyond the gates

code/scripts/make_demo.py run (the production API, model(points, calibration=...)) on every converted public frame of research/frnet/public_data/inputs, both profiles, against the research ORT runs (logs/frnet/m5_docs/demo/agreement.json):

set	frames	scored raw points	OT128 pooled (worst frame)	QT128 pooled (worst frame)
PandaSet 019 + 029, every 5th frame	32	3,442,737	99.66 % (99.35 %)	99.29 % (98.69 %)
nuScenes v1.0-mini mini_val, all key-frames	81	2,067,589	99.19 % (98.37 %)	99.44 % (98.10 %)

Scored rows equal the reference's on all 226 frame-profile pairs (same crop, same profile decisions); no frame was skipped. Mapped-label mIoU (research/frnet/public_data/scripts/frnet_public_eval.py, not a paper metric; logs/frnet/m5_docs/miou/): OT128 PandaSet coarse 0.7136 (p150) vs 0.7146 (CPU), fine 0.4471 vs 0.4479; nuScenes coarse 0.4827 vs 0.4831, fine 0.4515 vs 0.4531 (fine = mean over the classes with at least 1,000 labelled points, miou_classes_ge_1000_gt_points; over all labelled classes: PandaSet 0.3229 vs 0.3235, nuScenes 0.3285 vs 0.3292); QT128 PandaSet coarse 0.1869 vs 0.1873, nuScenes 0.3255 vs 0.3266.

Disclosed numerics changes

None: the model code, the precision knobs and the kernels are those of the baseline, and every existing gate value is bit-identical to it. The release changed what ships (samples), the tests (stricter) and the docs.

Review findings

• VERIFY_PORT round 1, low findings (all resolved or disclosed):
  1. stale PORT_LOG header and latency: fixed (ttaw 0.20.0 @ 89dec49; the superseded first-cut latency replaced by the current numbers; the duplicated known-issue paragraphs removed);
  2. test_public_frames skipped missing goldens silently: inside the workspace a missing golden, research frame or staged sample now FAILS (tests/_data.py need()), and the test asserts 17 / 5 frames;
  3. narrow precision margin (0.41 pt on nuScenes 0916 kf40): disclosed on the card (Caveats) and in OPT_REPORT.md as the binding constraint and backlog item 14;
  4. end-to-end latency depends on the host load: the card quotes the stage bench with its load conditions and the device rows separately;
  5. test_stage_pcc replays on its capture inputs: unchanged (hardening note); input rewriting is covered by test_small_graph, test_public_frames (17 / 5 frames through one trace per profile) and the new test_shipped_sample.
• New, low: on the wider (ungated) public set, 3 of 226 frame-profile pairs agree below 98.5 %: nuScenes scene-0916 key-frames 11 (98.374 %) and 1 (98.499 %) on OT128, key-frame 0 (98.105 %) on QT128. They are on the same out-of-domain scene (32-beam HDL-32E, Singapore) as the binding gated frame, are consistent with its 0.41-pt margin, and are disclosed on the card. They do not touch a frozen gate. The optimization phase should add them to the frames it judges precision changes on.
• Gates: no threshold was changed; the gate JSON gained only the four shipped-sample entries (0.985, the PLAN.md 2.8 argmax gate). No test compares TT with TT except replay == eager.
• Samples and data: code/ ships only the two synthetic scans with their presets and references (test_only_redistributable_samples_ship); no PandaSet, nuScenes or rosbag data is tracked; the PandaSet sample lives in the git-ignored staging_samples_pandaset/. nuScenes appears only as two renders labelled non-commercial.

p150 ETH-dispatch compliance

Every path that opens the device goes through ttaw.device.open_device, whose default is DispatchCoreConfig(ETH) (falling back to WORKER only with a RuntimeWarning, or never with allow_fallback=False):

path	how it opens	dispatch	evidence
FRNet.from_pretrained (API, quickstart, bench, profile, make_demo)	ModelBase.device_config -> DeviceConfig (DEVICE_DEFAULTS < FRNET_* env < args) -> open_device	eth default; FRNET_DISPATCH / dispatch= switch	every run printed dispatch eth, grid 12x10, fallback None
HTTP server (tt_frnet.server.app, the container)	the lifespan calls FRNet.from_pretrained with the env; serve.env pins FRNET_DISPATCH=eth, FRNET_NUM_CQS=1	eth (pinned; check_bundle.py refuses another value)	served runs: /info dispatch eth, grid 12x10; smoke_test.py fails on anything else
device tests (conftest.py device fixture)	tt_frnet.device.open_device(allow_fallback=False)	eth, no fallback	suite log
code/scripts/visualizer_report.py	open_device(allow_fallback=False)	eth	baseline log
code/scripts/diag_conv_precision.py, diag_frame_taps.py	tt_frnet.device.device_session()	eth default	code read
tests/test_tt_host.py	ttaw.device.open_device(0, dispatch="eth") on the fake ttnn only	– (no device)	host suite

The grid is never hard-coded: it comes from device.compute_with_storage_grid_size() (tt/layers.py, K1 segment.py); the only 12×10 literal is the expected-grid warning in ttaw/device.py and the smoke test's assertion. The image's verify: line asserts the patch marker in tt_metal/impl/dispatch/topology.cpp.

Not covered by this pass

• The container image: it is built and smoke-tested per serve profile (ot128, qt128) at packaging (code/scripts/container_smoke.sh), after this pass.
• A full adversarial verification of the release (that is VERIFY_PORT.md for the port; the next full verification follows the first optimization round).
• TensorRT parity: the reference is the fp32 ONNX Runtime run of the deployed ONNX, not Autoware's fp16 TensorRT engine.