score primitive never selects the first-listed option (0/300 ja, 0/290 en) — repro in NandhaKishorM/laya#131
Filed with full data as https://github.com/NandhaKishorM/laya/issues/131 — short version here for anyone using this checkpoint.
On 300 Japanese and 290 English label-conditioned emails, this checkpoint picks the first-listed score option 0–1 times out of ~300 under five conditions (original order, reversed, relabeled Low/Mid/High, relabeled reversed, 4 levels). When "Not urgent" is listed last instead of first, it is chosen 285 times. The mass follows the slot, not the label.
The English laya checkpoint does not show this (first option chosen 22–26% under the same conditions), so it is checkpoint-specific rather than a language or harness issue. In English this checkpoint also gets score RPS 0.340 (random 0.197) and bool AUROC 0.355.
Inference was laya.load() / agent.predict() as in the README, laya 0.3.4. Repro: https://github.com/hiroki-abe-58/sokudan/blob/main/docs/baseline_en.md
Practical note: ordinal questions usually list the lowest level first, so score on this checkpoint will rarely return the lowest level regardless of input.