Add erratum for the manifest ledger and git_dirty defects
Browse files- ERRATUM.md +73 -0
ERRATUM.md
ADDED
|
@@ -0,0 +1,73 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Erratum
|
| 2 |
+
|
| 3 |
+
Known defects in this archive, with what is affected and what to trust.
|
| 4 |
+
Recorded rather than quietly patched, because two of them cannot be corrected
|
| 5 |
+
without re-running the paid inference stage.
|
| 6 |
+
|
| 7 |
+
## 1. `run_manifest.json` embeds the wrong ledger (open)
|
| 8 |
+
|
| 9 |
+
**Affected:** `runs/full/provenance/run_manifest.json`, key `ledger`.
|
| 10 |
+
|
| 11 |
+
The full tier was produced by a real inference run and then re-invoked against
|
| 12 |
+
the warm logit cache. The re-invocation correctly declined to overwrite
|
| 13 |
+
`ledger_inference.json`, but still stamped its own near-zero ledger into the run
|
| 14 |
+
manifest, so the two disagree:
|
| 15 |
+
|
| 16 |
+
| | items_processed | throughput | USD | peak VRAM |
|
| 17 |
+
|---|---|---|---|---|
|
| 18 |
+
| `ledger_inference.json` (**authoritative**) | 903,558 | 2,808 views/s | $0.025 | 12.1 GB |
|
| 19 |
+
| `run_manifest.json` → `ledger` (**wrong**) | 0 | 0 views/s | ~$0.000004 | 0.57 GB |
|
| 20 |
+
|
| 21 |
+
**Trust `ledger_inference.json`.** Every figure and the reported compute table
|
| 22 |
+
are derived from it, not from the manifest copy.
|
| 23 |
+
|
| 24 |
+
The cause is fixed in source: `run_inference` now reads the preserved ledger
|
| 25 |
+
back and returns it, so the manifest, the printed summary and
|
| 26 |
+
`ledger_inference.json` cannot disagree in future runs. The stale manifest in
|
| 27 |
+
this archive is not regenerated because only the GPU stage writes it.
|
| 28 |
+
|
| 29 |
+
## 2. `git_dirty: true` is not meaningful here (open)
|
| 30 |
+
|
| 31 |
+
**Affected:** `run_manifest.json`, keys `git_commit` and `git_dirty`.
|
| 32 |
+
|
| 33 |
+
`_git_dirty()` calls `git status --porcelain` and reports true on any output,
|
| 34 |
+
including untracked files. The documented workflow creates `artifacts/split.json`
|
| 35 |
+
before inference and `.gitignore` does not exclude it, so `git_dirty` is true on
|
| 36 |
+
a clean checkout as well. The flag therefore does not distinguish "modified
|
| 37 |
+
tracked source" from "generated an expected artifact", and should not be read as
|
| 38 |
+
evidence that the producing source was modified.
|
| 39 |
+
|
| 40 |
+
`git_commit` records the upstream base commit, not a commit containing the
|
| 41 |
+
corrections applied at run time.
|
| 42 |
+
|
| 43 |
+
## 3. Figures and summaries were stale (fixed)
|
| 44 |
+
|
| 45 |
+
**Affected:** `runs/*/figures/*`, `runs/*/results/results_summary.md`.
|
| 46 |
+
|
| 47 |
+
`runs/full/figures/fig4_cost_throughput.png` previously showed 1.65e+07 views/s
|
| 48 |
+
and near-zero spend, taken from the ledger described in item 1 at a moment when
|
| 49 |
+
it was transiently wrong. The summaries additionally declared H1 from accuracy
|
| 50 |
+
alone and described the fitted model as three scalars rather than four.
|
| 51 |
+
|
| 52 |
+
All are regenerated in this revision. Figure 4 now shows 0.0894 GPU-h, $0.025,
|
| 53 |
+
2,808 views/s and 12.1 GB, and the summaries carry a per-metric H1 table and the
|
| 54 |
+
correct four-scalar count.
|
| 55 |
+
|
| 56 |
+
## 4. H1 does not hold strictly (finding, not a defect)
|
| 57 |
+
|
| 58 |
+
Accuracy, NLL, Brier and AURC all worsen monotonically with severity. **ECE does
|
| 59 |
+
not**: it dips at severities 3 to 4 and again from clean to severity 1. A paired
|
| 60 |
+
object-grouped bootstrap over base object IDs places both reversals within noise
|
| 61 |
+
of zero, so the direction of H1 is supported while strict monotonicity is not.
|
| 62 |
+
The summaries state this; earlier revisions did not.
|
| 63 |
+
|
| 64 |
+
## 5. Bootstrap depth deviates from the pre-registration (open)
|
| 65 |
+
|
| 66 |
+
`configs/full.yaml` runs **200** bootstrap replicates; the pre-registration
|
| 67 |
+
specified 1,000. The reduction was made because the CPU bootstrap dominated
|
| 68 |
+
runtime. `spec_hash` does not cover evaluation settings, so it is unchanged and
|
| 69 |
+
does not flag this. Deeper intervals can be recomputed from the cached logits
|
| 70 |
+
with no GPU.
|
| 71 |
+
|
| 72 |
+
Intervals on `delta_*_vs_clean` rows in `results.csv` are `NaN`: degradation
|
| 73 |
+
deltas are reported as point estimates only, without paired intervals.
|