| # Evaluation — B0 /root/models/Qwen3.8-27B |
|
|
| base: `/root/models/Qwen3.8-27B` · temperature: no · max_seq_len 1024 |
|
|
| ## Acceptance (DESIGN §8.3, main = test_set_30k) |
|
|
| **all rows** |
|
|
| | metric | target | value | B0 | pass | |
| |---|---|---|---|---| |
| | noul_auroc | >= 0.95 | 0.8763 | nan | ❌ | |
| | noul_brier_soft | <= 0.1 | 0.0628 | nan | ✅ | |
| | choice_top1 | >= 0.9 | 0.5814 | nan | ❌ | |
| | score_mae_expected | <= 0.35 | 0.8420 | nan | ❌ | |
| | kl | <= 0.15 | 0.4296 | nan | ❌ | |
| | ece | <= 0.03 | 0.0501 | nan | ❌ | |
| | ood KL ratio (ood/test_set_30k) | <= 2.0 | 1.525 | — | ✅ | |
|
|
| **excluding yuri_v1 exact-uniform placeholders (D1)** |
| |
| | metric | target | value | B0 | pass | |
| |---|---|---|---|---| |
| | noul_auroc | >= 0.95 | 0.8763 | nan | ❌ | |
| | noul_brier_soft | <= 0.1 | 0.0552 | nan | ✅ | |
| | choice_top1 | >= 0.9 | 0.5814 | nan | ❌ | |
| | score_mae_expected | <= 0.35 | 0.8420 | nan | ❌ | |
| | kl | <= 0.15 | 0.4411 | nan | ❌ | |
| | ece | <= 0.03 | 0.0307 | nan | ❌ | |
| | ood KL ratio (ood/test_set_30k) | <= 2.0 | 1.485 | — | ✅ | |
| |
| ## test_set_30k |
| |
| all rows: `n=29955 · kl=0.4296 · js=0.1105 · top1=0.5597 · ece=0.0501 · mce=0.2047 · noul_auroc=0.8763 · noul_brier_soft=0.0628 · noul_brier_hard=0.1580 · score_mae_expected=0.8420 · score_rps=0.3772 · choice_top1=0.5814 · choice_kl=0.4314` |
| |
| excluding yuri_v1 placeholders (2260 rows): `n=27695 · kl=0.4411 · js=0.1147 · top1=0.5597 · ece=0.0307 · mce=0.0532 · noul_auroc=0.8763 · noul_brier_soft=0.0552 · noul_brier_hard=0.1580 · score_mae_expected=0.8420 · score_rps=0.3772 · choice_top1=0.5814 · choice_kl=0.4314` |
| |
| throughput: 13918 tok/s · 130.5 rows/s |
| |
| ### test_set_30k by kind |
| |
| | kind | n | kl | top1 | ece | noul_auroc | noul_brier_soft | score_mae_expected | choice_top1 | |
| |---|---|---|---|---|---|---|---|---| |
| | choice | 9199 | 0.431 | 0.581 | 0.056 | — | — | — | 0.581 | |
| | noul | 12229 | 0.176 | 0.805 | 0.054 | 0.876 | 0.063 | — | — | |
| | score | 8527 | 0.791 | 0.253 | 0.051 | — | — | 0.842 | — | |
| |
| ### test_set_30k by source × kind |
| |
| | source | kind | n | kl | top1 | ece | noul_auroc | noul_brier_soft | score_mae_expected | choice_top1 | |
| |---|---|---|---|---|---|---|---|---|---| |
| | openjev_v2 | choice | 887 | 0.836 | 0.551 | 0.121 | — | — | — | 0.551 | |
| | openjev_v2 | noul | 1432 | 0.659 | 0.680 | 0.096 | 0.619 | 0.226 | — | — | |
| | yuri_v1 | noul | 2260 | 0.288 | — | 0.288 | — | 0.096 | — | — | |
| | yuri_v3 | choice | 8312 | 0.388 | 0.584 | 0.054 | — | — | — | 0.584 | |
| | yuri_v3 | noul | 8537 | 0.066 | 0.826 | 0.020 | 0.923 | 0.027 | — | — | |
| | yuri_v3 | score | 8527 | 0.791 | 0.253 | 0.051 | — | — | 0.842 | — | |
| |
| ### test_set_30k by family |
| |
| | family | n | kl | top1 | ece | noul_auroc | noul_brier_soft | score_mae_expected | choice_top1 | |
| |---|---|---|---|---|---|---|---|---| |
| | agent | 2583 | 0.393 | 0.543 | 0.035 | 0.921 | 0.026 | 0.937 | 0.570 | |
| | biology | 1654 | 0.384 | 0.562 | 0.048 | 0.967 | 0.023 | 0.780 | 0.624 | |
| | business | 2352 | 0.496 | 0.536 | 0.033 | 0.932 | 0.024 | 1.012 | 0.478 | |
| | chemistry | 1333 | 0.429 | 0.563 | 0.075 | 0.912 | 0.035 | 1.127 | 0.788 | |
| | genomics | 1914 | 0.420 | 0.559 | 0.057 | 0.913 | 0.027 | 0.746 | 0.635 | |
| | knowledge | 4807 | 0.342 | 0.538 | 0.157 | 0.897 | 0.076 | 0.674 | 0.514 | |
| | medical | 2120 | 0.442 | 0.586 | 0.024 | 0.954 | 0.030 | 0.917 | 0.566 | |
| | openjev | 2319 | 0.727 | 0.632 | 0.063 | 0.619 | 0.226 | — | 0.551 | |
| | physics | 1656 | 0.421 | 0.521 | 0.052 | 0.882 | 0.026 | 0.707 | 0.561 | |
| | science | 1667 | 0.338 | 0.596 | 0.019 | 0.909 | 0.023 | 0.657 | 0.632 | |
| | spatial | 1647 | 0.397 | 0.597 | 0.045 | 0.948 | 0.023 | 0.791 | 0.605 | |
| | structured | 1668 | 0.320 | 0.565 | 0.036 | 0.878 | 0.032 | 0.607 | 0.600 | |
| | technical | 2581 | 0.535 | 0.501 | 0.059 | 0.947 | 0.025 | 1.070 | 0.548 | |
| | theology | 1654 | 0.350 | 0.566 | 0.042 | 0.913 | 0.030 | 0.807 | 0.632 | |
| |
| ## ood |
| |
| all rows: `n=13058 · kl=0.6550 · js=0.1893 · top1=0.6902 · ece=0.0617 · mce=0.2023 · noul_auroc=0.6069 · noul_brier_soft=0.2071 · noul_brier_hard=0.2071 · score_mae_expected=0.6939 · score_rps=0.5406 · choice_top1=0.6356 · choice_kl=0.7858` |
| |
| excluding yuri_v1 placeholders (0 rows): `n=13058 · kl=0.6550 · js=0.1893 · top1=0.6902 · ece=0.0617 · mce=0.2023 · noul_auroc=0.6069 · noul_brier_soft=0.2071 · noul_brier_hard=0.2071 · score_mae_expected=0.6939 · score_rps=0.5406 · choice_top1=0.6356 · choice_kl=0.7858` |
| |
| throughput: 9876 tok/s · 37.3 rows/s |
| |
| ### ood by kind |
| |
| | kind | n | kl | top1 | ece | noul_auroc | noul_brier_soft | score_mae_expected | choice_top1 | |
| |---|---|---|---|---|---|---|---|---| |
| | choice | 3219 | 0.786 | 0.636 | 0.036 | — | — | — | 0.636 | |
| | noul | 9767 | 0.606 | 0.710 | 0.083 | 0.607 | 0.207 | — | — | |
| | score | 72 | 1.488 | 0.472 | 0.348 | — | — | 0.694 | — | |
| |
| ### ood by source × kind |
| |
| | source | kind | n | kl | top1 | ece | noul_auroc | noul_brier_soft | score_mae_expected | choice_top1 | |
| |---|---|---|---|---|---|---|---|---|---| |
| | openjev_v2 | choice | 3219 | 0.786 | 0.636 | 0.036 | — | — | — | 0.636 | |
| | openjev_v2 | noul | 9767 | 0.606 | 0.710 | 0.083 | 0.607 | 0.207 | — | — | |
| | openjev_v2 | score | 72 | 1.488 | 0.472 | 0.348 | — | — | 0.694 | — | |
| |
| ## Choice permutation consistency |
| |
| `{"n_rows": 500, "mean_max_abs_diff": 0.23575635254383087, "p90_max_abs_diff": 0.4639355540275574, "top1_flip_rate": 0.4095}` |
| |
