reports/b0_qwen38_27b.md

Evaluation — B0 /root/models/Qwen3.8-27B

base: /root/models/Qwen3.8-27B · temperature: no · max_seq_len 1024

Acceptance (DESIGN §8.3, main = test_set_30k)

all rows

metric	target	value	B0	pass
noul_auroc	>= 0.95	0.8763	nan	❌
noul_brier_soft	<= 0.1	0.0628	nan	✅
choice_top1	>= 0.9	0.5814	nan	❌
score_mae_expected	<= 0.35	0.8420	nan	❌
kl	<= 0.15	0.4296	nan	❌
ece	<= 0.03	0.0501	nan	❌
ood KL ratio (ood/test_set_30k)	<= 2.0	1.525	—	✅

excluding yuri_v1 exact-uniform placeholders (D1)

metric	target	value	B0	pass
noul_auroc	>= 0.95	0.8763	nan	❌
noul_brier_soft	<= 0.1	0.0552	nan	✅
choice_top1	>= 0.9	0.5814	nan	❌
score_mae_expected	<= 0.35	0.8420	nan	❌
kl	<= 0.15	0.4411	nan	❌
ece	<= 0.03	0.0307	nan	❌
ood KL ratio (ood/test_set_30k)	<= 2.0	1.485	—	✅

test_set_30k

all rows: n=29955 · kl=0.4296 · js=0.1105 · top1=0.5597 · ece=0.0501 · mce=0.2047 · noul_auroc=0.8763 · noul_brier_soft=0.0628 · noul_brier_hard=0.1580 · score_mae_expected=0.8420 · score_rps=0.3772 · choice_top1=0.5814 · choice_kl=0.4314

excluding yuri_v1 placeholders (2260 rows): n=27695 · kl=0.4411 · js=0.1147 · top1=0.5597 · ece=0.0307 · mce=0.0532 · noul_auroc=0.8763 · noul_brier_soft=0.0552 · noul_brier_hard=0.1580 · score_mae_expected=0.8420 · score_rps=0.3772 · choice_top1=0.5814 · choice_kl=0.4314

throughput: 13918 tok/s · 130.5 rows/s

test_set_30k by kind

kind	n	kl	top1	ece	noul_auroc	noul_brier_soft	score_mae_expected	choice_top1
choice	9199	0.431	0.581	0.056	—	—	—	0.581
noul	12229	0.176	0.805	0.054	0.876	0.063	—	—
score	8527	0.791	0.253	0.051	—	—	0.842	—

test_set_30k by source × kind

source	kind	n	kl	top1	ece	noul_auroc	noul_brier_soft	score_mae_expected	choice_top1
openjev_v2	choice	887	0.836	0.551	0.121	—	—	—	0.551
openjev_v2	noul	1432	0.659	0.680	0.096	0.619	0.226	—	—
yuri_v1	noul	2260	0.288	—	0.288	—	0.096	—	—
yuri_v3	choice	8312	0.388	0.584	0.054	—	—	—	0.584
yuri_v3	noul	8537	0.066	0.826	0.020	0.923	0.027	—	—
yuri_v3	score	8527	0.791	0.253	0.051	—	—	0.842	—

test_set_30k by family

family	n	kl	top1	ece	noul_auroc	noul_brier_soft	score_mae_expected	choice_top1
agent	2583	0.393	0.543	0.035	0.921	0.026	0.937	0.570
biology	1654	0.384	0.562	0.048	0.967	0.023	0.780	0.624
business	2352	0.496	0.536	0.033	0.932	0.024	1.012	0.478
chemistry	1333	0.429	0.563	0.075	0.912	0.035	1.127	0.788
genomics	1914	0.420	0.559	0.057	0.913	0.027	0.746	0.635
knowledge	4807	0.342	0.538	0.157	0.897	0.076	0.674	0.514
medical	2120	0.442	0.586	0.024	0.954	0.030	0.917	0.566
openjev	2319	0.727	0.632	0.063	0.619	0.226	—	0.551
physics	1656	0.421	0.521	0.052	0.882	0.026	0.707	0.561
science	1667	0.338	0.596	0.019	0.909	0.023	0.657	0.632
spatial	1647	0.397	0.597	0.045	0.948	0.023	0.791	0.605
structured	1668	0.320	0.565	0.036	0.878	0.032	0.607	0.600
technical	2581	0.535	0.501	0.059	0.947	0.025	1.070	0.548
theology	1654	0.350	0.566	0.042	0.913	0.030	0.807	0.632

ood

all rows: n=13058 · kl=0.6550 · js=0.1893 · top1=0.6902 · ece=0.0617 · mce=0.2023 · noul_auroc=0.6069 · noul_brier_soft=0.2071 · noul_brier_hard=0.2071 · score_mae_expected=0.6939 · score_rps=0.5406 · choice_top1=0.6356 · choice_kl=0.7858

excluding yuri_v1 placeholders (0 rows): n=13058 · kl=0.6550 · js=0.1893 · top1=0.6902 · ece=0.0617 · mce=0.2023 · noul_auroc=0.6069 · noul_brier_soft=0.2071 · noul_brier_hard=0.2071 · score_mae_expected=0.6939 · score_rps=0.5406 · choice_top1=0.6356 · choice_kl=0.7858

throughput: 9876 tok/s · 37.3 rows/s

ood by kind

kind	n	kl	top1	ece	noul_auroc	noul_brier_soft	score_mae_expected	choice_top1
choice	3219	0.786	0.636	0.036	—	—	—	0.636
noul	9767	0.606	0.710	0.083	0.607	0.207	—	—
score	72	1.488	0.472	0.348	—	—	0.694	—

ood by source × kind

source	kind	n	kl	top1	ece	noul_auroc	noul_brier_soft	score_mae_expected	choice_top1
openjev_v2	choice	3219	0.786	0.636	0.036	—	—	—	0.636
openjev_v2	noul	9767	0.606	0.710	0.083	0.607	0.207	—	—
openjev_v2	score	72	1.488	0.472	0.348	—	—	0.694	—

Choice permutation consistency

{"n_rows": 500, "mean_max_abs_diff": 0.23575635254383087, "p90_max_abs_diff": 0.4639355540275574, "top1_flip_rate": 0.4095}