# Evaluation — checkpoints/s2_9b_cooldown/best

base: `/root/models/Qwen3.5-9B` · temperature: yes · max_seq_len 1024

## Acceptance (DESIGN §8.3, main = test_set_30k)

**all rows**

| metric | target | value | B0 | pass |
|---|---|---|---|---|
| noul_auroc | >= 0.95 | 0.9938 | 0.8236 | ✅ |
| noul_brier_soft | <= 0.1 | 0.0022 | 0.0960 | ✅ |
| choice_top1 | >= 0.9 | 0.8836 | 0.5323 | ❌ |
| score_mae_expected | <= 0.35 | 0.1187 | 1.1297 | ✅ |
| kl | <= 0.15 | 0.0276 | 0.5100 | ✅ |
| ece | <= 0.03 | 0.0014 | 0.0942 | ✅ |
| ood KL ratio (ood/test_set_30k) | <= 2.0 | 7.526 | — | ❌ |

**excluding yuri_v1 exact-uniform placeholders (D1)**

| metric | target | value | B0 | pass |
|---|---|---|---|---|
| noul_auroc | >= 0.95 | 0.9938 | nan | ✅ |
| noul_brier_soft | <= 0.1 | 0.0026 | nan | ✅ |
| choice_top1 | >= 0.9 | 0.8836 | nan | ❌ |
| score_mae_expected | <= 0.35 | 0.1187 | nan | ✅ |
| kl | <= 0.15 | 0.0299 | nan | ✅ |
| ece | <= 0.03 | 0.0016 | nan | ✅ |
| ood KL ratio (ood/test_set_30k) | <= 2.0 | 6.966 | — | ❌ |

## test_set_30k

all rows: `n=29955 · kl=0.0276 · js=0.0072 · top1=0.9036 · ece=0.0014 · mce=0.0079 · noul_auroc=0.9938 · noul_brier_soft=0.0022 · noul_brier_hard=0.0761 · score_mae_expected=0.1187 · score_rps=0.0113 · choice_top1=0.8836 · choice_kl=0.0549`

excluding yuri_v1 placeholders (2260 rows): `n=27695 · kl=0.0299 · js=0.0078 · top1=0.9036 · ece=0.0016 · mce=0.0079 · noul_auroc=0.9938 · noul_brier_soft=0.0026 · noul_brier_hard=0.0761 · score_mae_expected=0.1187 · score_rps=0.0113 · choice_top1=0.8836 · choice_kl=0.0549`

throughput: 25870 tok/s · 242.6 rows/s

### test_set_30k by kind

| kind | n | kl | top1 | ece | noul_auroc | noul_brier_soft | score_mae_expected | choice_top1 |
|---|---|---|---|---|---|---|---|---|
| choice | 9199 | 0.055 | 0.884 | 0.003 | — | — | — | 0.884 |
| noul | 12229 | 0.006 | 0.959 | 0.002 | 0.994 | 0.002 | — | — |
| score | 8527 | 0.030 | 0.861 | 0.002 | — | — | 0.119 | — |

### test_set_30k by source × kind

| source | kind | n | kl | top1 | ece | noul_auroc | noul_brier_soft | score_mae_expected | choice_top1 |
|---|---|---|---|---|---|---|---|---|---|
| openjev_v2 | choice | 887 | 0.220 | 0.858 | 0.018 | — | — | — | 0.858 |
| openjev_v2 | noul | 1432 | 0.010 | 0.997 | 0.004 | 1.000 | 0.003 | — | — |
| yuri_v1 | noul | 2260 | 0.000 | — | 0.009 | — | 0.000 | — | — |
| yuri_v3 | choice | 8312 | 0.037 | 0.886 | 0.002 | — | — | — | 0.886 |
| yuri_v3 | noul | 8537 | 0.006 | 0.953 | 0.001 | 0.992 | 0.003 | — | — |
| yuri_v3 | score | 8527 | 0.030 | 0.861 | 0.002 | — | — | 0.119 | — |

### test_set_30k by family

| family | n | kl | top1 | ece | noul_auroc | noul_brier_soft | score_mae_expected | choice_top1 |
|---|---|---|---|---|---|---|---|---|
| agent | 2583 | 0.023 | 0.897 | 0.003 | 0.994 | 0.002 | 0.123 | 0.877 |
| biology | 1654 | 0.024 | 0.898 | 0.004 | 0.998 | 0.002 | 0.148 | 0.902 |
| business | 2352 | 0.022 | 0.930 | 0.003 | 0.997 | 0.002 | 0.089 | 0.904 |
| chemistry | 1333 | 0.020 | 0.931 | 0.005 | 0.997 | 0.002 | 0.113 | 0.946 |
| genomics | 1914 | 0.026 | 0.893 | 0.005 | 0.991 | 0.003 | 0.114 | 0.882 |
| knowledge | 4807 | 0.014 | 0.888 | 0.005 | 0.983 | 0.001 | 0.132 | 0.879 |
| medical | 2120 | 0.024 | 0.901 | 0.004 | 0.994 | 0.002 | 0.092 | 0.865 |
| openjev | 2319 | 0.090 | 0.945 | 0.007 | 1.000 | 0.003 | — | 0.858 |
| physics | 1656 | 0.023 | 0.881 | 0.004 | 0.992 | 0.002 | 0.117 | 0.869 |
| science | 1667 | 0.026 | 0.890 | 0.005 | 0.989 | 0.002 | 0.136 | 0.906 |
| spatial | 1647 | 0.025 | 0.886 | 0.007 | 0.984 | 0.002 | 0.119 | 0.850 |
| structured | 1668 | 0.025 | 0.885 | 0.004 | 0.983 | 0.004 | 0.152 | 0.881 |
| technical | 2581 | 0.025 | 0.918 | 0.004 | 0.993 | 0.002 | 0.089 | 0.878 |
| theology | 1654 | 0.025 | 0.893 | 0.005 | 0.992 | 0.002 | 0.144 | 0.909 |

## test

all rows: `n=14261 · kl=0.0252 · js=0.0065 · top1=0.9076 · ece=0.0029 · mce=0.0243 · noul_auroc=0.9948 · noul_brier_soft=0.0018 · noul_brier_hard=0.0653 · score_mae_expected=0.1165 · score_rps=0.0111 · choice_top1=0.8751 · choice_kl=0.0622`

excluding yuri_v1 placeholders (2951 rows): `n=11310 · kl=0.0317 · js=0.0082 · top1=0.9076 · ece=0.0014 · mce=0.0243 · noul_auroc=0.9948 · noul_brier_soft=0.0029 · noul_brier_hard=0.0653 · score_mae_expected=0.1165 · score_rps=0.0111 · choice_top1=0.8751 · choice_kl=0.0622`

throughput: 26259 tok/s · 204.4 rows/s

### test by kind

| kind | n | kl | top1 | ece | noul_auroc | noul_brier_soft | score_mae_expected | choice_top1 |
|---|---|---|---|---|---|---|---|---|
| choice | 3620 | 0.062 | 0.875 | 0.004 | — | — | — | 0.875 |
| noul | 7298 | 0.005 | 0.962 | 0.004 | 0.995 | 0.002 | — | — |
| score | 3343 | 0.030 | 0.872 | 0.003 | — | — | 0.116 | — |

### test by source × kind

| source | kind | n | kl | top1 | ece | noul_auroc | noul_brier_soft | score_mae_expected | choice_top1 |
|---|---|---|---|---|---|---|---|---|---|
| openjev_v2 | choice | 457 | 0.238 | 0.860 | 0.018 | — | — | — | 0.860 |
| openjev_v2 | noul | 1139 | 0.012 | 0.994 | 0.004 | 1.000 | 0.004 | — | — |
| yuri_v1 | noul | 2951 | 0.000 | — | 0.009 | — | 0.000 | — | — |
| yuri_v3 | choice | 3163 | 0.037 | 0.877 | 0.003 | — | — | — | 0.877 |
| yuri_v3 | noul | 3208 | 0.006 | 0.951 | 0.003 | 0.992 | 0.003 | — | — |
| yuri_v3 | score | 3343 | 0.030 | 0.872 | 0.003 | — | — | 0.116 | — |

## ood

all rows: `n=13058 · kl=0.2080 · js=0.0454 · top1=0.9164 · ece=0.0288 · mce=0.3191 · noul_auroc=0.9852 · noul_brier_soft=0.0427 · noul_brier_hard=0.0427 · score_mae_expected=0.7871 · score_rps=0.5324 · choice_top1=0.8397 · choice_kl=0.2989`

excluding yuri_v1 placeholders (0 rows): `n=13058 · kl=0.2080 · js=0.0454 · top1=0.9164 · ece=0.0288 · mce=0.3191 · noul_auroc=0.9852 · noul_brier_soft=0.0427 · noul_brier_hard=0.0427 · score_mae_expected=0.7871 · score_rps=0.5324 · choice_top1=0.8397 · choice_kl=0.2989`

throughput: 27200 tok/s · 102.6 rows/s

### ood by kind

| kind | n | kl | top1 | ece | noul_auroc | noul_brier_soft | score_mae_expected | choice_top1 |
|---|---|---|---|---|---|---|---|---|
| choice | 3219 | 0.299 | 0.840 | 0.044 | — | — | — | 0.840 |
| noul | 9767 | 0.170 | 0.945 | 0.024 | 0.985 | 0.043 | — | — |
| score | 72 | 1.278 | 0.444 | 0.236 | — | — | 0.787 | — |

### ood by source × kind

| source | kind | n | kl | top1 | ece | noul_auroc | noul_brier_soft | score_mae_expected | choice_top1 |
|---|---|---|---|---|---|---|---|---|---|
| openjev_v2 | choice | 3219 | 0.299 | 0.840 | 0.044 | — | — | — | 0.840 |
| openjev_v2 | noul | 9767 | 0.170 | 0.945 | 0.024 | 0.985 | 0.043 | — | — |
| openjev_v2 | score | 72 | 1.278 | 0.444 | 0.236 | — | — | 0.787 | — |

## validation

all rows: `n=14111 · kl=0.0260 · js=0.0067 · top1=0.9111 · ece=0.0030 · mce=0.0097 · noul_auroc=0.9943 · noul_brier_soft=0.0019 · noul_brier_hard=0.0656 · score_mae_expected=0.1196 · score_rps=0.0115 · choice_top1=0.8843 · choice_kl=0.0644`

excluding yuri_v1 placeholders (2920 rows): `n=11191 · kl=0.0327 · js=0.0084 · top1=0.9111 · ece=0.0020 · mce=0.0097 · noul_auroc=0.9943 · noul_brier_soft=0.0031 · noul_brier_hard=0.0656 · score_mae_expected=0.1196 · score_rps=0.0115 · choice_top1=0.8843 · choice_kl=0.0644`

throughput: 26303 tok/s · 204.2 rows/s

### validation by kind

| kind | n | kl | top1 | ece | noul_auroc | noul_brier_soft | score_mae_expected | choice_top1 |
|---|---|---|---|---|---|---|---|---|
| choice | 3592 | 0.064 | 0.884 | 0.005 | — | — | — | 0.884 |
| noul | 7318 | 0.005 | 0.963 | 0.004 | 0.994 | 0.002 | — | — |
| score | 3201 | 0.030 | 0.870 | 0.003 | — | — | 0.120 | — |

### validation by source × kind

| source | kind | n | kl | top1 | ece | noul_auroc | noul_brier_soft | score_mae_expected | choice_top1 |
|---|---|---|---|---|---|---|---|---|---|
| openjev_v2 | choice | 463 | 0.239 | 0.859 | 0.027 | — | — | — | 0.859 |
| openjev_v2 | noul | 1167 | 0.015 | 0.992 | 0.005 | 1.000 | 0.005 | — | — |
| yuri_v1 | noul | 2920 | 0.000 | — | 0.009 | — | 0.000 | — | — |
| yuri_v3 | choice | 3129 | 0.039 | 0.888 | 0.003 | — | — | — | 0.888 |
| yuri_v3 | noul | 3231 | 0.006 | 0.952 | 0.002 | 0.992 | 0.003 | — | — |
| yuri_v3 | score | 3201 | 0.030 | 0.870 | 0.003 | — | — | 0.120 | — |

## Choice permutation consistency

`{"n_rows": 1000, "mean_max_abs_diff": 0.030732551589608192, "p90_max_abs_diff": 0.06812191009521484, "top1_flip_rate": 0.04675}`