grKnight commited on
Commit
3563719
·
verified ·
1 Parent(s): 925ece8

Add erratum for the manifest ledger and git_dirty defects

Browse files
Files changed (1) hide show
  1. ERRATUM.md +73 -0
ERRATUM.md ADDED
@@ -0,0 +1,73 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Erratum
2
+
3
+ Known defects in this archive, with what is affected and what to trust.
4
+ Recorded rather than quietly patched, because two of them cannot be corrected
5
+ without re-running the paid inference stage.
6
+
7
+ ## 1. `run_manifest.json` embeds the wrong ledger (open)
8
+
9
+ **Affected:** `runs/full/provenance/run_manifest.json`, key `ledger`.
10
+
11
+ The full tier was produced by a real inference run and then re-invoked against
12
+ the warm logit cache. The re-invocation correctly declined to overwrite
13
+ `ledger_inference.json`, but still stamped its own near-zero ledger into the run
14
+ manifest, so the two disagree:
15
+
16
+ | | items_processed | throughput | USD | peak VRAM |
17
+ |---|---|---|---|---|
18
+ | `ledger_inference.json` (**authoritative**) | 903,558 | 2,808 views/s | $0.025 | 12.1 GB |
19
+ | `run_manifest.json` → `ledger` (**wrong**) | 0 | 0 views/s | ~$0.000004 | 0.57 GB |
20
+
21
+ **Trust `ledger_inference.json`.** Every figure and the reported compute table
22
+ are derived from it, not from the manifest copy.
23
+
24
+ The cause is fixed in source: `run_inference` now reads the preserved ledger
25
+ back and returns it, so the manifest, the printed summary and
26
+ `ledger_inference.json` cannot disagree in future runs. The stale manifest in
27
+ this archive is not regenerated because only the GPU stage writes it.
28
+
29
+ ## 2. `git_dirty: true` is not meaningful here (open)
30
+
31
+ **Affected:** `run_manifest.json`, keys `git_commit` and `git_dirty`.
32
+
33
+ `_git_dirty()` calls `git status --porcelain` and reports true on any output,
34
+ including untracked files. The documented workflow creates `artifacts/split.json`
35
+ before inference and `.gitignore` does not exclude it, so `git_dirty` is true on
36
+ a clean checkout as well. The flag therefore does not distinguish "modified
37
+ tracked source" from "generated an expected artifact", and should not be read as
38
+ evidence that the producing source was modified.
39
+
40
+ `git_commit` records the upstream base commit, not a commit containing the
41
+ corrections applied at run time.
42
+
43
+ ## 3. Figures and summaries were stale (fixed)
44
+
45
+ **Affected:** `runs/*/figures/*`, `runs/*/results/results_summary.md`.
46
+
47
+ `runs/full/figures/fig4_cost_throughput.png` previously showed 1.65e+07 views/s
48
+ and near-zero spend, taken from the ledger described in item 1 at a moment when
49
+ it was transiently wrong. The summaries additionally declared H1 from accuracy
50
+ alone and described the fitted model as three scalars rather than four.
51
+
52
+ All are regenerated in this revision. Figure 4 now shows 0.0894 GPU-h, $0.025,
53
+ 2,808 views/s and 12.1 GB, and the summaries carry a per-metric H1 table and the
54
+ correct four-scalar count.
55
+
56
+ ## 4. H1 does not hold strictly (finding, not a defect)
57
+
58
+ Accuracy, NLL, Brier and AURC all worsen monotonically with severity. **ECE does
59
+ not**: it dips at severities 3 to 4 and again from clean to severity 1. A paired
60
+ object-grouped bootstrap over base object IDs places both reversals within noise
61
+ of zero, so the direction of H1 is supported while strict monotonicity is not.
62
+ The summaries state this; earlier revisions did not.
63
+
64
+ ## 5. Bootstrap depth deviates from the pre-registration (open)
65
+
66
+ `configs/full.yaml` runs **200** bootstrap replicates; the pre-registration
67
+ specified 1,000. The reduction was made because the CPU bootstrap dominated
68
+ runtime. `spec_hash` does not cover evaluation settings, so it is unchanged and
69
+ does not flag this. Deeper intervals can be recomputed from the cached logits
70
+ with no GPU.
71
+
72
+ Intervals on `delta_*_vs_clean` rows in `results.csv` are `NaN`: degradation
73
+ deltas are reported as point estimates only, without paired intervals.