VERIFICATION_2026-10-09.md

# frnet-p150: independent verification, 2026-10-09

## Verdict

**PASS.** This is the light verification pass of the first publish (`research/PLAN.md` §6.3): the frozen gates
re-run, the baseline numbers re-checked, and every device-open path audited for ETH dispatch. It was made by the agent
that wrote the release docs, which is neither the porter, nor the adversarial verifier of `VERIFY_PORT.md`, nor the
baseline agent, on the release-code commit `04c39b5` (the model code of the baseline `0dd52b2`; the shipped sample
became a synthetic scan per serve profile, the PandaSet sample is staged and not shipped, test hardening; the docs
commit that follows changes `make_demo.py`'s render captions and the docs only). The device runs ran on 2026-10-09
15:53-16:10 UTC in one `bin/devrun` window (a first queueing attempt died while waiting for the lock, when the
workspace's `bin/devrun` was edited under it; it had not touched the device). Against the baseline commit `0dd52b2`:

- **Frozen gates:** the device suite (`test_pcc_device.py` + `test_e2e_device.py`, one pytest) passed **14 / 14**
  tests. Every existing gate passes with values **bit-identical** to the baseline run (the `_m5` and `_m5docs` reports:
  2,144 leaves compared, 0 differ; timings excluded), including the binding frame: OT128 min agreement 98.907 %
  (nuScenes 0916 kf40) against the 98.5 % gate. The two new tests (`test_shipped_sample`, and `test_public_frames`
  now requiring all 17 / 5 frames) passed; the four new shipped-sample gates were frozen at this first green run
  (label agreement OT128 0.99550, QT128 0.99623; filtered-cloud agreement 0.99772 / 0.99872; threshold 0.985).
- **Baseline re-check:** the device rows of `OPT_BASELINE.md` repeat to within 0.03 ms (OT128 210.02 / 209.95 ms
  trace / b2b on the synthetic sample vs 210.00 / 209.95 ms now; QT128 133.53 / 133.43 vs 133.50 / 133.43 ms). Host
  rows within the load noise (OT128 `model()` 292.6 -> 289.3 ms p50 on the synthetic sample, 355.7 -> 360.1 ms on
  PandaSet, at load average 4.1).
- **Shipped samples:** the shipped `synthetic_<variant>.npz` are byte-identical to the baseline's synthetic bench
  inputs (`cmp`), so the baseline's synthetic rows are the shipped sample's. p150 vs the stored CPU reference:
  OT128 99.550 % of 52,613 points, QT128 99.623 % of 69,553, the same through `bench.py`, the device test, the
  quickstart, `make_demo.py` and the served smoke test.
- **Public data (113 frames per profile, production API):** pooled class agreement OT128 PandaSet 99.66 % / nuScenes
  99.19 %, QT128 99.29 % / 99.44 %; identical scored rows to the CPU reference on all 226 frame-profile pairs, no
  profile skip. 3 pairs fall below 98.5 % (not gated; see "Review findings"). Mapped-label mIoU on the p150 equals the
  CPU reference's within 0.002.
- **ETH dispatch:** every device-open path opens ETH / 1 CQ / 12×10 by default; every run of this pass printed
  `dispatch eth`, `grid 12x10`, `cores 120`, `fallback null`, `eth_patch true` (both serve profiles, API and server).
- **No numerics change** and no new bug: nothing in the model code was changed by this pass.

## Setup

| item | value |
|---|---|
| Hardware | one Blackhole p150b, ETH dispatch, 1 CQ, 12×10 compute grid (120 cores), AICLK 1350 MHz median (min 1343) during the benches |
| tt-metal | `44d66500520` with `patches/tt-metal-eth-dispatch.patch` (sha256 `08d0ddf6…`, byte-identical to the workspace copy) |
| Weights | `AutowareFoundation/lidar_frnet` @ `a53a1c11b0b28d31fd50a03abef5db0e830a5b3c` (tag `v2.0`): the device suite and the demo run read the sha256-pinned workspace copy; the quickstart, the served runs and the load-time runs resolved the HF cache at the pinned revision offline (downloaded without a token for this pass; every file identical to the workspace copy, ONNX sha256 ot128 `d29bf595…`, qt128 `647af927…`) |
| Verified commit | `04c39b5` (ttaw 0.20.0, `common` `89dec49`, `source_dirty: false`) |
| Baseline commit | `0dd52b2` (`OPT_BASELINE.md`) |
| Workload | the shipped samples `code/tt_frnet/samples/synthetic_{ot128,qt128}.npz` and the staged PandaSet 019 f40 (107,146 points, local), batch 1, warm; the 17 / 5 gate frames; 113 converted public frames per profile (local only) |
| Bench method | `code/scripts/bench.py --iters 100 --warmup 10 --b2b-iters 50 --b2b-rounds 5`, one run per serve profile (the baseline's settings), with each output checked against its stored CPU reference |
| Served method | uvicorn on the host with the serve pins of `tt-model.yaml` (the container's app; `FRNET_VARIANT` per profile), the bundle's `smoke_test.py` against it, then 5 warm-up + 50 requests of the shipped sample from a loopback client, the README's `client.py` + `curl` recipe, a knobs request and a request without calibration |
| Accuracy reference | the fp32 CPU reference (`code/tt_frnet/reference/`, ONNX Runtime on the deployed ONNX; VERIFY_PORT: within 1e-11 of an independent ORT run); for the 113-frame sets the research ORT runs (`research/frnet/public_data/runs`) |
| Host | AMD EPYC VM, 8 vCPUs, shared with other agents' jobs (load average 4.1-6.1 during the runs) |
| Logs | `logs/frnet/m5_docs/` of the workspace (job scripts in `scripts/`) |

## Measured baseline vs re-check

All times in ms, p50 of 100 iterations (b2b: 5 rounds of 50 replays). Baseline: `OPT_BASELINE.md` (`0dd52b2`);
re-check: `code/scripts/bench.py` with the baseline's settings on `04c39b5`
(`logs/frnet/m5_docs/bench_recheck_{ot128,qt128}.json`).

| row | baseline | re-check |
|---|---:|---:|
| OT128, shipped sample: trace, one replay / back-to-back | 210.02 / 209.948 | 210.00 / 209.946 |
| OT128, PandaSet 019 f40: trace / back-to-back | 211.72 / 211.643 | 211.73 / 211.639 |
| QT128, shipped sample: trace / back-to-back | 133.53 / 133.434 | 133.50 / 133.429 |
| QT128, PandaSet 019 f40: trace / back-to-back | 134.91 / 134.866 | 134.92 / 134.856 |
| OT128 H2D / D2H, shipped sample | 5.42 / 7.65 | 5.42 / 7.56 |
| OT128 host pre-processing, shipped sample / PandaSet | 55.3 / 107.2 | 54.7 / 107.4 |
| OT128 `model()` p50, shipped sample / PandaSet | 292.6 / 355.7 | 289.3 / 360.1 |
| QT128 `model()` p50, shipped sample / PandaSet | 221.1 / 244.3 | 205.4 / 241.5 |
| served `/predict` `timing_ms.total`, OT128 / QT128 (median of 50) | – | 310.4 / 228.9 |
| served client round trip, loopback | – | 328.1 / 247.3 |
| `from_pretrained`, empty JIT cache / warm cache, OT128 | – / 6.5 s | 156.6 / 7.5 s |
| `from_pretrained`, empty JIT cache / warm cache, QT128 | – / 6.9 s | 151.9 / 6.6 s |

The served and cold-load rows are new for the card (the baseline measured neither). The first two calls after a load
took 320.0 / 304.7 ms (OT128) and 241.1 / 214.9 ms (QT128): the trace is captured during the load. The cold JIT cache
holds 682 MB (OT128) / 644 MB (QT128) after the load.

## Accuracy gate results

`logs/frnet/m5_docs/suite.log`; reports `logs/frnet/{pcc,pcc_small,e2e,latency}_{ot128,qt128}_m5docs.json`.

| check | gate | OT128 | QT128 |
|---|---|---|---|
| every gated module tap (PCC vs the fp32 reference, PandaSet sample, replay outputs) | ≥ 0.999 | min 0.999729 (`point_fusion1`) | min 0.999649 (`attention4`) |
| `pred_probs` PCC / raw-point agreement, PandaSet sample (staged) | ≥ 0.998 / ≥ 98.5 % | 0.999942 / 99.581 % | 0.999844 / 99.295 % |
| public frames (17 / 5): min `pred_probs` PCC | ≥ 0.998 | 0.999688 | 0.999805 |
| public frames: min raw-point agreement | ≥ 98.5 % | **98.907 %** (nuScenes 0916 kf40) | 99.158 % (PandaSet 029 f40) |
| shipped sample vs its stored reference body: labels / filtered-cloud mask (new) | ≥ 98.5 % | 99.550 % / 99.772 % | 99.623 % / 99.872 % |
| replay == eager | bit-exact | 45 / 45 taps | 45 / 45 taps |
| API == `/predict` handler | identical | yes | yes |

Host suite (fake ttnn, gates read-only): **117 passed, 15 skipped** (the 14 device tests and the onnxruntime-only
test), `logs/frnet/m5_docs/host_suite.log`. Research venv (ONNX Runtime + torch): **85 passed, 2 skipped** (by design),
`logs/frnet/m5_docs/research_venv.log`. The counts differ from VERIFY_PORT's (112 / 13 host, 83 / 2 research venv) by the tests the release added or
changed (release hygiene of the shipped samples, the top-level `PYTHON.md` copy, the shipped-sample device test).

### Public-frame agreement beyond the gates

`code/scripts/make_demo.py run` (the production API, `model(points, calibration=...)`) on every converted public frame
of `research/frnet/public_data/inputs`, both profiles, against the research ORT runs
(`logs/frnet/m5_docs/demo/agreement.json`):

| set | frames | scored raw points | OT128 pooled (worst frame) | QT128 pooled (worst frame) |
|---|---:|---:|---:|---:|
| PandaSet 019 + 029, every 5th frame | 32 | 3,442,737 | 99.66 % (99.35 %) | 99.29 % (98.69 %) |
| nuScenes v1.0-mini mini_val, all key-frames | 81 | 2,067,589 | 99.19 % (98.37 %) | 99.44 % (98.10 %) |

Scored rows equal the reference's on all 226 frame-profile pairs (same crop, same profile decisions); no frame was
skipped. Mapped-label mIoU (`research/frnet/public_data/scripts/frnet_public_eval.py`, not a paper metric;
`logs/frnet/m5_docs/miou/`): OT128 PandaSet coarse 0.7136 (p150) vs 0.7146 (CPU), fine 0.4471 vs 0.4479; nuScenes
coarse 0.4827 vs 0.4831, fine 0.4515 vs 0.4531 (fine = mean over the classes with at least 1,000 labelled points,
`miou_classes_ge_1000_gt_points`; over all labelled classes: PandaSet 0.3229 vs 0.3235, nuScenes 0.3285 vs 0.3292); QT128 PandaSet coarse 0.1869 vs 0.1873, nuScenes 0.3255 vs 0.3266.

## Disclosed numerics changes

None: the model code, the precision knobs and the kernels are those of the baseline, and every existing gate value is
bit-identical to it. The release changed what ships (samples), the tests (stricter) and the docs.

## Review findings

- **VERIFY_PORT round 1, low findings** (all resolved or disclosed):
  1. stale PORT_LOG header and latency: fixed (ttaw 0.20.0 @ `89dec49`; the superseded first-cut latency replaced by
     the current numbers; the duplicated known-issue paragraphs removed);
  2. `test_public_frames` skipped missing goldens silently: inside the workspace a missing golden, research frame or
     staged sample now FAILS (`tests/_data.py need()`), and the test asserts 17 / 5 frames;
  3. narrow precision margin (0.41 pt on nuScenes 0916 kf40): disclosed on the card (Caveats) and in `OPT_REPORT.md`
     as the binding constraint and backlog item 14;
  4. end-to-end latency depends on the host load: the card quotes the stage bench with its load conditions and the
     device rows separately;
  5. `test_stage_pcc` replays on its capture inputs: unchanged (hardening note); input rewriting is covered by
     `test_small_graph`, `test_public_frames` (17 / 5 frames through one trace per profile) and the new `test_shipped_sample`.
- **New, low:** on the wider (ungated) public set, 3 of 226 frame-profile pairs agree below 98.5 %: nuScenes
  scene-0916 key-frames 11 (98.374 %) and 1 (98.499 %) on OT128, key-frame 0 (98.105 %) on QT128. They are on the same
  out-of-domain scene (32-beam HDL-32E, Singapore) as the binding gated frame, are consistent with its 0.41-pt margin,
  and are disclosed on the card. They do not touch a frozen gate. The optimization phase should add them to the
  frames it judges precision changes on.
- **Gates:** no threshold was changed; the gate JSON gained only the four shipped-sample entries (0.985, the PLAN.md 2.8
  argmax gate). No test compares TT with TT except replay == eager.
- **Samples and data:** `code/` ships only the two synthetic scans with their presets and references
  (`test_only_redistributable_samples_ship`); no PandaSet, nuScenes or rosbag data is tracked; the PandaSet sample
  lives in the git-ignored `staging_samples_pandaset/`. nuScenes appears only as two renders labelled non-commercial.

## p150 ETH-dispatch compliance

Every path that opens the device goes through `ttaw.device.open_device`, whose default is
`DispatchCoreConfig(ETH)` (falling back to WORKER only with a `RuntimeWarning`, or never with `allow_fallback=False`):

| path | how it opens | dispatch | evidence |
|---|---|---|---|
| `FRNet.from_pretrained` (API, quickstart, bench, profile, make_demo) | `ModelBase.device_config` -> `DeviceConfig` (`DEVICE_DEFAULTS` < `FRNET_*` env < args) -> `open_device` | `eth` default; `FRNET_DISPATCH` / `dispatch=` switch | every run printed `dispatch eth, grid 12x10, fallback None` |
| HTTP server (`tt_frnet.server.app`, the container) | the lifespan calls `FRNet.from_pretrained` with the env; `serve.env` pins `FRNET_DISPATCH=eth`, `FRNET_NUM_CQS=1` | `eth` (pinned; `check_bundle.py` refuses another value) | served runs: `/info` `dispatch eth`, `grid 12x10`; `smoke_test.py` fails on anything else |
| device tests (`conftest.py` `device` fixture) | `tt_frnet.device.open_device(allow_fallback=False)` | `eth`, no fallback | suite log |
| `code/scripts/visualizer_report.py` | `open_device(allow_fallback=False)` | `eth` | baseline log |
| `code/scripts/diag_conv_precision.py`, `diag_frame_taps.py` | `tt_frnet.device.device_session()` | `eth` default | code read |
| `tests/test_tt_host.py` | `ttaw.device.open_device(0, dispatch="eth")` on the fake ttnn only | – (no device) | host suite |

The grid is never hard-coded: it comes from `device.compute_with_storage_grid_size()` (`tt/layers.py`, K1
`segment.py`); the only 12×10 literal is the expected-grid warning in `ttaw/device.py` and the smoke test's assertion.
The image's `verify:` line asserts the patch marker in `tt_metal/impl/dispatch/topology.cpp`.

## Not covered by this pass

- The container image: it is built and smoke-tested per serve profile (`ot128`, `qt128`) at packaging
  (`code/scripts/container_smoke.sh`), after this pass.
- A full adversarial verification of the release (that is `VERIFY_PORT.md` for the port; the next full verification
  follows the first optimization round).
- TensorRT parity: the reference is the fp32 ONNX Runtime run of the deployed ONNX, not Autoware's fp16 TensorRT engine.