Draw three tables the repository shipped but never showed

Where the parameters live: cutting 24L to 4L removes 84% of the layer stack and does not touch the embedding, so the embedding's share climbs from 33.8% to 76.1% — which is why the vocabulary was the next cut. At 39,866 entries it is back to 33.8%, the teacher's own balance.

The ladder's losses: not monotone as totals (0.176 / 0.034 / 0.047) and monotone rising per layer removed (0.0110 / 0.0169 / 0.0235). The last two layers cost 2.1x per layer what the first sixteen did.

Thresholds: the frozen grid selects a threshold that moves up to 4x between seeds of one arm, and every run over-predicts — 34.4% of pairs are positive, predictions land at 41-71%, and one control run returns 1.34 false positives per true one. Added with the advice not to inherit these thresholds.

README_KO.md mirrors all three. SHA256SUMS: 358 -> 362 entries.

.gitattributes

@@ -28,3 +28,6 @@ benchmark/figures/16_field_depth_effect.png filter=lfs diff=lfs merge=lfs -text
 benchmark/figures/17_field_resolution_floor.png filter=lfs diff=lfs merge=lfs -text
 benchmark/figures/18_vocabulary_cost_inversion.png filter=lfs diff=lfs merge=lfs -text
 benchmark/figures/19_window_cost.png filter=lfs diff=lfs merge=lfs -text
+benchmark/figures/20_where_the_parameters_live.png filter=lfs diff=lfs merge=lfs -text
+benchmark/figures/21_ladder_loss.png filter=lfs diff=lfs merge=lfs -text
+benchmark/figures/22_threshold_and_overprediction.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -218,6 +218,30 @@ root. If you want this vocabulary, start from this root.
 
 ---
 
+
+### If you train a head, the threshold is yours to pick
+
+One thing the benchmark's own tables show clearly, and this card had not: the decision threshold
+the protocol lands on is unstable, and at whatever it lands on the head over-predicts.
+
+![threshold and over-prediction](benchmark/figures/22_threshold_and_overprediction.png)
+
+Across three seeds of one arm the frozen 18-point grid selected 0.10, 0.15 and 0.30, and for
+another it selected 0.05, 0.20 and 0.20 — a 3x and a 4x range for one model on one split, from
+nothing but the seed. The task-blind arms behave the same way: across the thirteen task-blind runs
+of the final test the selected threshold spans 0.05 to 0.45.
+
+And every run predicts more positives than the data contains. 34.4% of the (document, label)
+pairs are actually positive; the selected thresholds put predictions at 41% to 71%. One control
+run returns 1.34 false positives for every true one. That is what a threshold picked to maximise
+macro F1 on a 14-label problem with rare labels does — macro F1 rewards getting *something* right
+on the rare labels, and the cheapest way to do that is to say yes more often.
+
+None of this is a defect in the backbone; it is a property of the frozen protocol's selection
+rule and of the split's label sparsity. It is here because it is the part of these numbers most
+likely to surprise you when you attach your own head: **do not inherit these thresholds, and
+calibrate on your own data for the precision/recall balance your application wants.**
+
 ## Architecture
 
 ```
@@ -246,6 +270,38 @@ the map, that choice is in the weights. Cutting the vocabulary does not remove i
 
 ---
 
+
+### Where the parameters actually live
+
+The two cuts had to happen in that order, and the reason is arithmetic rather than judgement.
+
+![where the parameters live](benchmark/figures/20_where_the_parameters_live.png)
+
+At 24 layers the embedding table is a third of the model. Cutting to 4 layers removes 84% of the
+layer stack — 498M parameters down to 80M — and touches the embedding not at all, so the model
+only falls from 752M to 334M and the embedding's share climbs from 33.8% to **76.1%**. After the
+depth cut, three quarters of what remained was a lookup table for 248,320 entries that this model
+would never need, which is what made the vocabulary the next thing worth cutting.
+
+Cutting it to 39,866 brings the embedding back to 33.8% of the model — the same balance the
+24-layer teacher had — and the model to 120.6M. The two grid points below this root keep going:
+at 15,380 entries the embedding is 16% and the layer stack is nearly everything left.
+
+### What the ladder itself cost
+
+Distillation is three stages, and the tables in this repository record each one's loss.
+
+![the ladder's losses](benchmark/figures/21_ladder_loss.png)
+
+Read as totals the sequence is not monotone: 24→8 drops sixteen layers for 0.176, 8→6 drops two
+for 0.034, and then 6→4 drops two for 0.047 — 39% more than the step before it for the same
+number of layers. Read per layer removed it is monotone and rising: 0.0110, 0.0169, 0.0235. The
+last two layers cost 2.1x per layer what the first sixteen did.
+
+We do not read a mechanism into that. It is one run per stage with no spread, on a matching
+objective that never sees a label, and the interface loss is larger than the final loss at every
+stage, which is what one would expect when the boundaries being matched are the constraint.
+
 ## The path, step by step
 
 Everything below is measured. Each section is a **situation we assumed**, and it names the data
@@ -840,7 +896,7 @@ described at the top of this card.
 | [`root_manifest.json`](root_manifest.json) | its vocabulary rule, layer maps, hashes, why this grid point |
 | [`benchmark/full/`](benchmark/full/) | the 61-arm table, the grid, the same-host cuts |
 | [`benchmark/cost_probe/`](benchmark/cost_probe/) | the tokenisation-cost and window-cost measurements, scripts and results |
-| [`benchmark/figures/`](benchmark/figures/) | the nineteen figures above |
+| [`benchmark/figures/`](benchmark/figures/) | the twenty-two figures above |
 | [`runtime/`](runtime/) | the two relaxed sources of the dedicated runtime, and its own README |
 | [`docker/`](docker/) | a container that runs the backbone with a head on it, CPU only |
 | [`docker-train/`](docker-train/) | train a classifier on your own data, convert, serve |

README_KO.md

@@ -194,6 +194,30 @@ qwen35-distill finetune \
 
 ---
 
+
+### 헤드를 학습시키신다면, 임계값은 직접 고르셔야 합니다
+
+벤치마크 자신의 표가 분명히 보여 주는데 이 카드가 말하지 않은 것이 하나 있습니다. 프로토콜이
+고르는 결정 임계값은 불안정하고, 무엇에 안착하든 헤드는 과다예측합니다.
+
+![임계값과 과다예측](benchmark/figures/22_threshold_and_overprediction.png)
+
+한 arm의 세 시드에서 동결된 18점 그리드는 0.10, 0.15, 0.30을 골랐고, 다른 arm에서는 0.05, 0.20,
+0.20을 골랐습니다. 한 모델, 한 분할에서 오직 시드만 달랐는데 3배와 4배의 범위입니다. task-blind
+arm들도 같습니다. 최종 테스트의 task-blind 실행 열세 개에서 선택된 임계값은 0.05에서 0.45까지
+퍼져 있습니다.
+
+그리고 모든 실행이 데이터에 있는 것보다 더 많은 양성을 예측합니다. (문서, 레이블) 쌍의 34.4%가
+실제 양성인데, 선택된 임계값들은 예측을 41%에서 71%에 놓습니다. 한 대조 실행은 참 양성 하나당 거짓
+양성 1.34개를 돌려줍니다. 드문 레이블이 있는 14개 레이블 문제에서 macro F1을 최대화하도록 고른
+임계값이 하는 일이 이것입니다 — macro F1은 드문 레이블에서 *무언가*를 맞히는 것을 보상하고, 그
+가장 싼 방법은 더 자주 예라고 말하는 것입니다.
+
+이 중 어느 것도 백본의 결함이 아니라 동결된 프로토콜의 선택 규칙과 이 분할의 레이블 희소성의
+성질입니다. 이것을 여기 적는 이유는, 여러분이 자신의 헤드를 붙이실 때 이 수치들 중 가장 놀라기
+쉬운 부분이기 때문입니다. **이 임계값들을 그대로 물려받지 마시고, 여러분의 응용이 원하는
+정밀도/재현율 균형에 맞게 여러분의 데이터에서 캘리브레이션하십시오.**
+
 ## 아키텍처
 
 ```
@@ -222,6 +246,37 @@ macro F1 기준 1위입니다. 이 저장소의 예전 `config.json` 파일들
 
 ---
 
+
+### 파라미터는 실제로 어디에 있는가
+
+두 번의 절단이 그 순서여야 했던 이유는 판단이 아니라 산수입니다.
+
+![파라미터가 있는 곳](benchmark/figures/20_where_the_parameters_live.png)
+
+24계층에서 임베딩 테이블은 모델의 3분의 1입니다. 4계층으로 자르면 계층 스택의 84% — 498M에서
+80M으로 — 가 사라지지만 임베딩은 전혀 건드려지지 않으므로, 모델은 752M에서 334M으로만 줄고
+임베딩의 비중은 33.8%에서 **76.1%**로 올라갑니다. 깊이를 자르고 나니 남은 것의 4분의 3이 이 모델이
+결코 쓰지 않을 248,320개 항목의 참조표였고, 그래서 다음에 자를 가치가 있는 것이 어휘였습니다.
+
+39,866으로 자르면 임베딩은 모델의 33.8%로 돌아오는데, 이는 24계층 교사가 가지고 있던 것과 같은
+균형이며, 모델은 120.6M이 됩니다. 이 루트 아래의 두 그리드 포인트는 더 내려갑니다. 15,380 항목에서는
+임베딩이 16%이고 남은 것의 거의 전부가 계층 스택입니다.
+
+### 사다리 자체는 무엇을 치렀는가
+
+증류는 세 단계이고, 이 저장소의 표가 각 단계의 손실을 기록하고 있습니다.
+
+![사다리의 손실](benchmark/figures/21_ladder_loss.png)
+
+총합으로 읽으면 순서가 단조롭지 않습니다. 24→8은 열여섯 계층을 0.176에, 8→6은 두 계층을 0.034에,
+그리고 6→4는 두 계층을 0.047에 — 같은 계층 수에 대해 바로 앞 단계보다 39% 더 — 떨어뜨립니다.
+제거된 계층당으로 읽으면 단조롭게 올라갑니다. 0.0110, 0.0169, 0.0235. 마지막 두 계층은 처음 열여섯
+계층이 치른 값의 계층당 2.1배를 치릅니다.
+
+여기에서 기전을 읽어 내지는 않습니다. 단계당 한 번의 실행이라 산포가 없고, 레이블을 한 번도 보지
+않는 정합 목적함수이며, 모든 단계에서 인터페이스 손실이 최종 손실보다 큰데 이는 정합 대상이 되는
+경계가 제약일 때 예상되는 모습입니다.
+
 ## 경로, 단계별로
 
 아래의 모든 것은 측정된 값입니다. 각 절은 **우리가 가정한 상황**이며, 그 상황을 세우기 위해 사용한
@@ -805,7 +860,7 @@ A/B/A, 스레드 수 1/3/4, 라이브 서비스 경로도 모두 일치했습니
 | [`root_manifest.json`](root_manifest.json) | 그 어휘 규칙, 계층 맵, 해시, 왜 이 그리드 포인트인지 |
 | [`benchmark/full/`](benchmark/full/) | 61개 arm 표, 그리드, 동일 호스트 절단 |
 | [`benchmark/cost_probe/`](benchmark/cost_probe/) | 토크나이즈 비용과 윈도 비용 측정 스크립트와 결과 |
-| [`benchmark/figures/`](benchmark/figures/) | 위의 열아홉 개 그림 |
+| [`benchmark/figures/`](benchmark/figures/) | 위의 스물두 개 그림 |
 | [`runtime/`](runtime/) | 전용 런타임의 완화된 두 소스와 그 자신의 README |
 | [`docker/`](docker/) | 헤드를 얹�� 백본을 돌리는 컨테이너, CPU 전용 |
 | [`docker-train/`](docker-train/) | 자기 데이터로 분류기를 학습시키고, 변환하고, 서빙 |

SHA256SUMS

@@ -1,6 +1,6 @@
 77fd4710def9ec3c0f6225800e0235f15a425abd4a8b03559127fcd782612049  LICENSE
-9cf91e7d56c96ed3901391cdb4ea14e43e78eeaaf7b647b8ae59ae2811e7a02b  README.md
-30d3e1ed49635d513c27e5b615c425d0c56bfb591cfef2a112bcaf1ef7a11d08  README_KO.md
+92ea56cfedc17d67ad2dcd4fa3d40378ac6a8ed27456bb24721284030e3349b3  README.md
+a62448e225d5b10ec7a8bc998be7cc7db209ba59f1ff6ce66e4b2293ddbaa2b6  README_KO.md
 bf4073de38c3905effd2dcce6f38894f325c8a0a79bedfb2779f0545c86b9c9f  benchmark/BENCHMARK_CARD.md
 e01bd7ac59a3212e44b77e5fcfbc7c41ec91d74c998306d2438209d6ef1a3aa7  benchmark/arm_summary.csv
 2c544c3907c4f592d724f658850b8c6b9634845cb5b50a66ebd2ea6c5d48ce0b  benchmark/compression_ladder.csv
@@ -39,6 +39,9 @@ cfb3071db4077bed7aee06c141f5f12873779a732516242580c889fb223c14fa  benchmark/figu
 dc621e266a694496ba30d9d5c03e89e4d6c65616f5c25df0216d012102cd3f53  benchmark/figures/17_field_resolution_floor.png
 8941a6f6bcf3a354d5a7b3b53367c36508a2aed73c6094df7536401d41e0cfd7  benchmark/figures/18_vocabulary_cost_inversion.png
 261f2fddb02eb02d2d7d696d650405cea153941eb6e93073bf21483732f9db0e  benchmark/figures/19_window_cost.png
+87c6e141f8f28c293382e3198efe360fc4a4d6bdd42fc5b73bc2eb5a170eb5a3  benchmark/figures/20_where_the_parameters_live.png
+8dc8781f83f6c3de22b287b86db544da986e0225345db5e43dd77e54ab4b1443  benchmark/figures/21_ladder_loss.png
+b08436d88b8ad64dccebf9d04629fd1dfbd1e5f21e0908f2a5284f4b7243642d  benchmark/figures/22_threshold_and_overprediction.png
 7c238ed969ed64ac7c63084c663ecc531b618622199a77b09d28f461db11a657  benchmark/figures/README.md
 afd203646513c49140a25ef2a562a064286a4fa5efb495e6a73f83f0106e7933  benchmark/figures/cross_task/05_why_4l_edge_tradeoff.csv
 22f531ea586f9c79ac8f114345cec56e74ea5ed97aa3726f7310018666ae6312  benchmark/figures/cross_task/05_why_4l_edge_tradeoff.pdf
@@ -54,6 +57,7 @@ c3312e98b8abbbdc488e5dfde77c4110732e8d08abe9ed931bd985ad799ecad8  benchmark/figu
 d42a85f828b719a6b24e226013ae660f3ff88280b256490b9eb08e19616c96d7  benchmark/figures/make_cost_figures.py
 3f6b1b82af1619afab8c753cd836040a7353a3226646b8b99fb641617e4ecec7  benchmark/figures/make_more_figures.py
 5f6b1979dcb7ff4ee897659b2f902dd3cc36339987f42f681bff7be5ae06d304  benchmark/figures/make_readout_figures.py
+8acf0dcd72bb77dc650aa8253db35de2d31df0a67ef20f928075f82297242283  benchmark/figures/make_structure_figures.py
 078b4f34897e0a1b9bcff66ce21ba0feb28d72c86eaa8cef608f110c48f52f2e  benchmark/figures/v128k_compression.csv
 014da12a435e3404b31470e2642d180dc111eb608fb8513fd993097019698b7a  benchmark/figures/v128k_quality_efficiency.csv
 644e5b7469c0b4e0b63d5ecaa1c0f2fc8e07af8bb5d9e4d3ae5a88166de374f3  benchmark/figures/v128k_quality_storage.png

benchmark/figures/make_structure_figures.py

@@ -0,0 +1,234 @@
+"""Three figures from tables the card shipped but never drew.
+
+20  where the parameters actually live, and why the vocabulary was cut after the layers
+21  the distillation ladder's own losses: the difficulty is not monotone
+22  the threshold the protocol selects moves between seeds, and every arm over-predicts
+
+Run from benchmark/figures/ . Reads ../compression_ladder.csv and ../seed_metrics.csv,
+both of which ship in this repository.
+"""
+import csv
+import json
+import os
+import statistics as st
+
+import matplotlib
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+from matplotlib.lines import Line2D
+from matplotlib.patches import Patch
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+BENCH = os.path.join(HERE, "..") + os.sep        # benchmark/ holds the CSVs this reads
+OUT = HERE + os.sep
+
+INK, MUTED, FAINT, GRID = "#1a1a1a", "#6a6a72", "#9a9aa2", "#e6e6ea"
+ROOTC, GRIDC, QWENC, ENCC, DOWN = "#1a5496", "#0d7a52", "#b05512", "#9a9aa2", "#c0392b"
+H = 1024                       # hidden size, unchanged down the whole ladder
+
+
+def style(ax, xgrid=True, ygrid=True):
+    ax.grid(axis="both" if (xgrid and ygrid) else ("x" if xgrid else "y"), color=GRID, lw=.8)
+    ax.set_axisbelow(True)
+    for s in ("top", "right"):
+        ax.spines[s].set_visible(False)
+    for s in ("bottom", "left"):
+        ax.spines[s].set_color("#c9c9cf")
+    ax.tick_params(length=0, colors="#4a4a52")
+
+
+def footer(fig, text):
+    fig.text(.006, .012, text, fontsize=8.6, color=MUTED, ha="left")
+
+
+def fig_parameters():
+    """Two cuts in sequence: depth first, then vocabulary. Embedding share is the story."""
+    ladder = list(csv.DictReader(open(BENCH + "compression_ladder.csv")))
+    emb248 = 248320 * H
+    depth = [(f"{r['layers']}L", int(r["parameters"]), emb248) for r in ladder]
+    layer_at_4l = depth[-1][1] - emb248
+    vocab = [("248,320", 248320), ("128,000", 128000), ("72,455", 72455),
+             ("39,866", 39866), ("23,551", 23551), ("15,380", 15380)]
+    vocab = [(lab, v * H + layer_at_4l + 1024, v * H) for lab, v in vocab]
+
+    fig, (ax, ax2) = plt.subplots(1, 2, figsize=(14.6, 6.4))
+    fig.patch.set_facecolor("white")
+
+    for axis, data, xlab, title, sub in (
+            (ax, depth, "layers kept  (vocabulary held at 248,320)",
+             "Cut 1 — depth: 24 layers to 4",
+             "the layer stack loses 84% of its parameters and the embedding does not move"),
+            (ax2, vocab, "vocabulary entries kept  (depth held at 4 layers)",
+             "Cut 2 — vocabulary: 248,320 entries to 39,866",
+             "by then the embedding was three quarters of the model \u2014 which is why it came second")):
+        labs = [d[0] for d in data]
+        tot = [d[1] / 1e6 for d in data]
+        embm = [d[2] / 1e6 for d in data]
+        lays = [t - e for t, e in zip(tot, embm)]
+        axis.bar(labs, embm, width=.62, color=ROOTC, zorder=3, label="embedding table")
+        axis.bar(labs, lays, bottom=embm, width=.62, color=ENCC, zorder=3, label="layer stack")
+        for i, (t, e) in enumerate(zip(tot, embm)):
+            axis.text(i, t + max(tot) * .035, f"{t:,.0f}M", ha="center", fontsize=9.2,
+                      color=INK, fontweight="bold")
+            axis.text(i, e / 2, f"{e/t:.0%}", ha="center", va="center", fontsize=9,
+                      color="white", fontweight="bold")
+        axis.set_xlabel(xlab, fontsize=10)
+        axis.set_ylabel("parameters (millions)", fontsize=10)
+        axis.set_ylim(0, max(tot) * 1.24)
+        style(axis, xgrid=False)
+        axis.set_title(f"{title}\n{sub}", fontsize=11.5, loc="left", color=INK, pad=12)
+
+    # mark the two endpoints that actually shipped
+    ax.get_xticklabels()[-1].set_color(ROOTC)
+    ax.get_xticklabels()[-1].set_fontweight("bold")
+    ax2.get_xticklabels()[3].set_color(ROOTC)
+    ax2.get_xticklabels()[3].set_fontweight("bold")
+    ax2.annotate("this root", (3, vocab[3][1] / 1e6), textcoords="offset points",
+                 xytext=(0, 52), ha="center", fontsize=9.6, color=ROOTC, fontweight="bold",
+                 arrowprops=dict(arrowstyle="-|>", color=ROOTC, lw=1.3,
+                                 shrinkA=2, shrinkB=22))
+    ax.legend(handles=[Patch(color=ROOTC, label="embedding table"),
+                       Patch(color=ENCC, label="layer stack")],
+              fontsize=9, frameon=False, loc="upper right")
+
+    footer(fig, "Percentages are the embedding's share of the model. Cutting depth alone raised that "
+                "share from 33.8% to 76.1%; cutting the vocabulary brought it back to 33.8%, the "
+                "same balance the 24-layer teacher had. Hidden size is 1,024 throughout.")
+    fig.tight_layout(rect=(0, .055, 1, 1))
+    fig.savefig(OUT + "20_where_the_parameters_live.png", dpi=170, facecolor="white")
+    plt.close(fig)
+
+
+def fig_ladder_loss():
+    rows = [r for r in csv.DictReader(open(BENCH + "compression_ladder.csv")) if r["kd_stage"]]
+    stages = [r["kd_stage"].replace("to", " → ") for r in rows]
+    tot = [float(r["mean_total_loss"]) for r in rows]
+    itf = [float(r["mean_interface_loss"]) for r in rows]
+    fin = [float(r["mean_final_loss"]) for r in rows]
+    dropped = [16, 2, 2]
+
+    fig, (ax, ax2) = plt.subplots(1, 2, figsize=(13.8, 5.8),
+                                  gridspec_kw={"width_ratios": [1.12, 1]})
+    fig.patch.set_facecolor("white")
+
+    x = range(len(stages))
+    w = .26
+    for off, vals, c, lab in ((-w, itf, ROOTC, "interface loss"),
+                              (0.0, tot, GRIDC, "total loss"),
+                              (w, fin, QWENC, "final-layer loss")):
+        ax.bar([i + off for i in x], vals, width=w * .92, color=c, zorder=3, label=lab)
+        for i, v in zip(x, vals):
+            ax.text(i + off, v + .004, f"{v:.3f}", ha="center", fontsize=8.4,
+                    color=INK, fontweight="bold")
+    ax.set_xticks(list(x))
+    ax.set_xticklabels([f"{s}\n{d} layers removed" for s, d in zip(stages, dropped)], fontsize=9.4)
+    ax.set_ylabel("mean KD loss over the distillation run", fontsize=10)
+    ax.set_ylim(0, max(itf) * 1.22)
+    style(ax, xgrid=False)
+    ax.legend(fontsize=9, frameon=False, loc="upper right", ncol=1)
+    ax.set_title("The ladder's own losses are not monotone\n"
+                 "24\u21928 drops 16 layers for 0.176; 8\u21926 drops 2 for 0.034; then 6\u21924 "
+                 "drops 2 for 0.047",
+                 fontsize=11.5, loc="left", color=INK, pad=12)
+
+    # per-layer-removed difficulty
+    per = [t / d for t, d in zip(tot, dropped)]
+    ax2.plot(list(x), per, "-o", color=ROOTC, lw=2.4, ms=11, mec="white", mew=1.7, zorder=4)
+    for i, v in zip(x, per):
+        ax2.annotate(f"{v:.4f}", (i, v), textcoords="offset points", xytext=(0, 14),
+                     ha="center", fontsize=9.2, color=INK, fontweight="bold")
+    ax2.set_xticks(list(x))
+    ax2.set_xticklabels(stages, fontsize=9.6)
+    ax2.set_xlim(-.45, len(stages) - .55)
+    ax2.set_ylabel("total loss per layer removed", fontsize=10)
+    ax2.set_ylim(0, max(per) * 1.30)
+    style(ax2, xgrid=False)
+    ax2.set_title("Per layer removed, each step is harder than the last\n"
+                  "the final two layers cost 2.1x per layer what the first sixteen did",
+                  fontsize=11.5, loc="left", color=INK, pad=12)
+
+    footer(fig, "Interface loss matches hidden states at the kept-layer boundaries; final loss matches "
+                "the last hidden state. No logits and no labels enter either. One run per stage, so "
+                "there is no spread to report and no significance is claimed.")
+    fig.tight_layout(rect=(0, .055, 1, 1))
+    fig.savefig(OUT + "21_ladder_loss.png", dpi=170, facecolor="white")
+    plt.close(fig)
+
+
+def fig_thresholds():
+    rows = list(csv.DictReader(open(BENCH + "seed_metrics.csv")))
+    NAMES = {"task_agnostic_base": "task-free 4L root",
+             "structural_copy_control": "structural copy control",
+             "existing_specialized_separate_lineage": "specialised 4L, separate lineage"}
+    arms = list(dict.fromkeys(r["arm"] for r in rows))
+    actual = [(int(r["tp"]) + int(r["fn"])) / (55 * 14) for r in rows]
+    assert max(actual) - min(actual) < 1e-9
+    actual = actual[0]
+
+    fig, (ax, ax2) = plt.subplots(1, 2, figsize=(14.2, 5.8))
+    fig.patch.set_facecolor("white")
+
+    colours = {a: c for a, c in zip(arms, (ROOTC, QWENC, GRIDC))}
+    for a in arms:
+        rs = [r for r in rows if r["arm"] == a]
+        th = [float(r["threshold"]) for r in rs]
+        pp = [float(r["predicted_positive_rate"]) for r in rs]
+        ax.plot(th, pp, "o", ms=11, color=colours[a], mec="white", mew=1.6, zorder=4)
+        for r, t, p in zip(rs, th, pp):
+            ax.annotate(f"seed {r['seed']}", (t, p), textcoords="offset points",
+                        xytext=(11, -3), fontsize=8.2, color=MUTED)
+    ax.axhline(actual, color=DOWN, ls=(0, (5, 3)), lw=1.6, zorder=3)
+    ax.text(.328, actual + .006, f"actual positive rate {actual:.3f}",
+            fontsize=8.8, color=DOWN, ha="right", va="bottom")
+    ax.set_xlabel("threshold the frozen grid selected for that run", fontsize=10)
+    ax.set_ylabel("fraction of (document, label) pairs predicted positive", fontsize=10)
+    ax.set_xlim(.02, .335)
+    ax.set_ylim(.335, .775)
+    style(ax)
+    ax.legend(handles=[Line2D([], [], marker="o", ls="", color=colours[a], ms=9, label=NAMES[a])
+                       for a in arms], fontsize=8.8, frameon=False, loc="upper right")
+    ax.set_title("Every run predicts more positives than the data has\n"
+                 "and the threshold the grid selects moves up to 4x between seeds of one arm",
+                 fontsize=11.5, loc="left", color=INK, pad=12)
+
+    labels, tps, fps, fns, cols = [], [], [], [], []
+    for a in arms:
+        for r in [r for r in rows if r["arm"] == a]:
+            labels.append(f"{NAMES[a].split(',')[0]}\nseed {r['seed']}  thr {r['threshold']}")
+            tps.append(int(r["tp"]))
+            fps.append(int(r["fp"]))
+            fns.append(int(r["fn"]))
+            cols.append(colours[a])
+    y = range(len(labels))
+    ax2.barh(list(y), tps, height=.6, color=ROOTC, zorder=3, label="true positives")
+    ax2.barh(list(y), fps, left=tps, height=.6, color=QWENC, zorder=3, label="false positives")
+    ax2.barh(list(y), fns, left=[t + f for t, f in zip(tps, fps)], height=.6,
+             color=ENCC, zorder=3, label="false negatives")
+    for i, (t, f) in enumerate(zip(tps, fps)):
+        ax2.text(t + f + 12, i, f"{f/t:.2f} fp per tp", va="center", fontsize=8.4,
+                 color=DOWN if f > t else MUTED, fontweight="bold" if f > t else "normal")
+    ax2.set_yticks(list(y))
+    ax2.set_yticklabels(labels, fontsize=7.8)
+    ax2.set_xlabel("(document, label) pairs, out of 770", fontsize=10)
+    ax2.set_xlim(0, 720)
+    ax2.invert_yaxis()
+    style(ax2, xgrid=True, ygrid=False)
+    ax2.spines["left"].set_visible(False)
+    ax2.legend(fontsize=8.6, frameon=False, loc="lower right", ncol=1)
+    ax2.set_title("What that costs, run by run\n"
+                  "one control run returns more false positives than true ones",
+                  fontsize=11.5, loc="left", color=INK, pad=12)
+
+    footer(fig, "Opened 56-document calibration split, 55 scored articles x 14 labels. If you train a "
+                "head on this backbone, expect to pick the threshold yourself: the protocol's own grid "
+                "lands on a different one for every seed.")
+    fig.tight_layout(rect=(0, .055, 1, 1))
+    fig.savefig(OUT + "22_threshold_and_overprediction.png", dpi=170, facecolor="white")
+    plt.close(fig)
+
+
+if __name__ == "__main__":
+    os.makedirs(OUT, exist_ok=True)
+    for fn in (fig_parameters, fig_ladder_loss, fig_thresholds):
+        fn()
+        print("ok", fn.__name__)