Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,242 @@
+---
+license: apache-2.0
+base_model: Qwen/Qwen3.5-35B-A3B-Base
+language: [en]
+pipeline_tag: text-classification
+tags: [decision-model, calibrated, structured-output, multi-task, system-one, one-pass, mixture-of-experts]
+---
+
+# decider-35b-a3b: typed decisions with calibrated probabilities in one forward pass, 35B mixture of experts
+
+A language model that does not generate text. It reads a state and one or more typed questions, each with an explicit option
+list, and returns a probability distribution over the options for every question from one forward pass. There is no decoding,
+no parsing and no output outside the options you defined. It is called from software, not chatted with. It is an open
+reproduction of the "System One" model class (TypeSafe AI's Jev).
+
+Base model: [Qwen/Qwen3.5-35B-A3B-Base](https://huggingface.co/Qwen/Qwen3.5-35B-A3B-Base): 34.7B parameters, of which 3B
+are active per token (256 routed experts, 8 active, plus a shared expert; 40 layers, 10 with full attention and 30 with
+gated delta-net linear attention). The supervised recipe of [decider-2b](https://huggingface.co/Mapika/decider-2b) (one epoch of
+cross-entropy on the slot readout over the public decision mixture) was applied to it with the routed experts frozen and the
+Muon optimizer on the block matrices. **This repository holds v1**, the bf16 weights (65 GB). An NVFP4 checkpoint of the
+same weights for vLLM and TensorRT-LLM is at
+[Mapika/decider-35b-a3b-nvfp4](https://huggingface.co/Mapika/decider-35b-a3b-nvfp4). Code, data registry, training
+scripts and the recipe are at https://github.com/Mapika/decider; `decider/` in this repository is the inference subset of that
+package.
+
+Against decider-2b v10 on the same rows: accuracy is higher on 93 of the 95 regression tasks (in-task 0.855 against 0.805,
+held-out 0.810 against 0.755), +6.7 points on the 847 validation rows, +5.0 on OpenJev, +6.9 on Mind2Web, +5.9 on the
+TypeSafe workflow rows, JevBench hard tier 0.676 against 0.459, Bespoke's public suite 0.774 against 0.704 macro. Negative
+log-likelihood drops by 0.12 to 0.24 nats on every fixture. The model was not RL-trained: on live browser tasks its greedy
+play beats v10 (97.2% against 90.9%) and its sampled play is behind (86.4% against 93.2%). Details under Evaluation.
+
+## Usage
+
+```python
+from decider.infer import Decider          # decider/ is included in this repo
+d = Decider("Mapika/decider-35b-a3b", use_graphs=False)
+d.decide("My card was charged twice for the same purchase.",
+         [{"question": "Which department should handle this?", "options": ["billing", "technical support", "sales"]},
+          {"question": "Does this need a refund action?", "options": ["no", "yes"]}])
+# [{'choice': 'billing', 'confidence': 0.99, 'probs': {...}}, {'choice': 'yes', 'confidence': 0.98, 'probs': {...}}]
+```
+
+The API is the same as decider-2b's: `decide_batch` scores many states with many questions in one call, `abstain_below=t`
+returns `None` under a confidence threshold, a question can have 2 to 255 options, and `system_one` / `decider.serve` accept
+TypeSafe's `POST /v1/systemone` request shape (the official `typesafe-sdk` works with `TYPESAFE_BASE_URL` pointing at the
+server). Every question and every Score level is scored in its own row. The state may be a string, object or array of up to
+32k tokens. See the decider-2b card for the full description of the request shape, field types and the schema cache.
+
+Requirements: one GPU with at least 80 GB of memory (the weights take 65 GB in bf16), `torch>=2.14`, `transformers>=5.17`
+and `flash-linear-attention`. `config.json` sets `experts_implementation: grouped_mm`, which runs the 256 experts of a layer
+as one grouped matrix multiplication; the eager expert loop that `transformers` falls back to on older versions is about
+13x slower. `use_graphs=False` is required: the CUDA-graph engine and the FP8 path of the helper package were built for the
+dense models and are untested with this architecture. Loading takes about 25 seconds from local disk.
+
+Without the helper package, the same computation in plain `transformers`:
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+tok = AutoTokenizer.from_pretrained(REPO); m = AutoModelForCausalLM.from_pretrained(REPO, dtype=torch.bfloat16).cuda().eval()
+prompt = ("Context:\nMy card was charged twice for the same purchase.\n\n"
+          "Question: Which department should handle this?\nOptions:\n(A) billing\n(B) technical support\n(C) sales\nAnswer: (")
+ids = tok(prompt, return_tensors="pt").to("cuda")
+with torch.no_grad():
+    logits = m(**ids).logits[0, -1]
+letters = [tok.encode(L, add_special_tokens=False)[0] for L in "ABC"]
+probs = torch.softmax(logits[letters].float() / 1.08, -1)      # 1.08 is the stored temperature
+```
+
+## How it works
+
+The prompt is `Context: ...` followed by, for each question, the question text, the lettered options `(A) ... (B) ...` and an
+answer slot `Answer k: (`. The hidden state at each slot is projected with the option-letter rows of the LM head and softmaxed
+over the valid letters, divided by the temperature in `decider_config.json`. Letters are never generated, so all slots are read
+from one pass. Large label sets were sub-sampled to at most 10 options per training example (gold always kept, order shuffled),
+so the model conditions on the supplied candidates rather than on a fixed head.
+
+## Training
+
+One epoch of the public supervised mixture of the GitHub repository (`scripts/train.sh full`: about 95 public decision
+datasets, agent trajectories, Mind2Web element choice, game states, teacher-written custom questions, Jev's input shapes,
+two prompt layouts, isolated Score levels, 10% abstention rows). 1,543,567 items, 463M tokens, pre-tokenized once and read in
+the same order by every rank.
+
+| | |
+|---|---|
+| trainable parameters | 2.45B: attention, delta-net, shared experts, routers, norms, embeddings, LM head. The 256 routed experts of every layer (32.2B parameters) stay at the base weights. |
+| optimizer | Muon on the 250 block matrices (1.41B parameters): momentum 0.95, Nesterov, 5 Newton-Schulz steps, update scaled by 0.2 sqrt(max(rows, cols)) so the AdamW learning-rate schedule applies. AdamW (betas 0.9 / 0.95) on embeddings, head, norms, routers, shared-expert gates, convolutions and 1-D parameters (1.04B). FP32 master weights, bf16 forward. |
+| schedule | peak learning rate 1e-5, 150 warm-up steps, cosine to zero, 16,287 steps of 32,768 tokens, gradient clip 1.0, no weight decay |
+| hardware | 4 NVIDIA B300, data parallel, 8,192-token micro-batches per GPU, gradient checkpointing, grouped-GEMM experts; 394 minutes at 22,000 to 25,000 tokens per second, 100 GB peak per GPU |
+| training cross-entropy | 0.93 over the first 200 steps, 0.50 at 25%, 0.46 at 50%, 0.43 over the last 300 steps |
+
+Muon was chosen over AdamW on a same-data comparison stopped at 11% of the epoch: at every logged step both optimizers had
+seen identical examples, and Muon's cross-entropy was lower in 75 of 94 logged windows, 0.537 against 0.605 at step 1,880
+(mean over steps 1,500 to 1,880: 0.557 against 0.614). No AdamW model was trained to the end, so there is no AdamW row in the
+evaluation tables. The temperature 1.08 was fitted on the in-task half of the regression set. No reinforcement-learning stage
+was run on this model; the RL recipe of decider-2b v10 is documented in `docs/RL.md` of the GitHub repository.
+
+## Evaluation
+
+**Public regression set**, rebuilt on this machine (95 tasks: 67 in-task, 28 held-out; large label sets sub-sampled to 10
+options; one temperature per model fitted on in-task data). The decider-2b rows are the same set, same rows. ECE is the
+expected calibration error with 15 bins.
+
+| model | in-task acc / NLL / ECE (67 tasks) | held-out acc / NLL / ECE (28 tasks) |
+|---|---|---|
+| Qwen3.5-35B-A3B-Base, zero-shot, T=1.27 | 0.732 / 0.693 / 0.096 | 0.749 / 0.650 / 0.082 |
+| after 25% of the epoch, T=0.96 | 0.839 / 0.399 / 0.032 | 0.803 / 0.522 / 0.073 |
+| after 50%, T=1.08 | 0.850 / 0.370 / 0.028 | 0.813 / 0.488 / 0.064 |
+| **decider-35b-a3b v1 (this repository), T=1.08** | **0.855 / 0.357 / 0.026** | **0.810 / 0.497 / 0.069** |
+| decider-2b v10, T=1.30 | 0.805 / 0.474 / 0.037 | 0.755 / 0.622 / 0.084 |
+| decider-2b v8, T=1.30 | 0.806 / 0.473 / 0.038 | 0.757 / 0.622 / 0.083 |
+
+Half the epoch reaches 99% of the final in-task accuracy; held-out accuracy is flat from 50% to 100%. Accuracy is above v10 on
+93 of the 95 tasks and 0.6 points below on two (counterfactual detection, offensive-tweet detection). The largest gains are
+on knowledge and reasoning tasks: MedQA +31 points, MedMCQA +24, TruthfulQA +22, Winogrande +20, MMLU +19, StrategyQA +19.
+
+Per-task accuracy / ECE on the held-out datasets, v10 against this model:
+
+| task | decider-2b v10 | decider-35b-a3b |
+|---|---|---|
+| abstain_probe | 0.606 / 0.134 | 0.622 / 0.085 |
+| ade | 0.817 / 0.038 | 0.837 / 0.035 |
+| arena_pref | 0.483 / 0.189 | 0.521 / 0.121 |
+| bbc_news | 0.927 / 0.013 | 0.944 / 0.027 |
+| cb | 0.857 / 0.093 | 0.893 / 0.084 |
+| cr_reviews | 0.903 / 0.031 | 0.914 / 0.032 |
+| dbpedia_l2 | 0.950 / 0.018 | 0.961 / 0.010 |
+| dbpedia_l3 | 0.987 / 0.005 | 0.992 / 0.004 |
+| dolly_category | 0.299 / 0.203 | 0.354 / 0.098 |
+| fin_phrasebank | 0.694 / 0.042 | 0.759 / 0.110 |
+| fin_sentiment | 0.793 / 0.058 | 0.839 / 0.136 |
+| hermes_tools | 0.723 / 0.208 | 0.799 / 0.085 |
+| hwu64 | 0.961 / 0.030 | 0.975 / 0.022 |
+| massive_scenario | 0.756 / 0.041 | 0.799 / 0.028 |
+| offtopic_probe | 0.841 / 0.027 | 0.870 / 0.038 |
+| paws | 0.724 / 0.145 | 0.729 / 0.168 |
+| pubmedqa | 0.756 / 0.085 | 0.820 / 0.078 |
+| quality | 0.494 / 0.233 | 0.632 / 0.096 |
+| quality_full | 0.508 / 0.198 | 0.565 / 0.112 |
+| reward_bench | 0.819 / 0.045 | 0.919 / 0.024 |
+| sciq | 0.982 / 0.024 | 0.993 / 0.011 |
+| social_iqa | 0.708 / 0.077 | 0.823 / 0.025 |
+| strategyqa | 0.552 / 0.138 | 0.739 / 0.036 |
+| student_questions | 0.925 / 0.045 | 0.954 / 0.090 |
+| trec | 0.784 / 0.066 | 0.832 / 0.160 |
+| truthfulqa | 0.537 / 0.090 | 0.754 / 0.068 |
+| tweet_irony | 0.795 / 0.052 | 0.861 / 0.129 |
+| xstory_cloze | 0.962 / 0.017 | 0.995 / 0.016 |
+
+**On the same rows as decider-2b.** Every row below is scored by both models on identical inputs and seeds. Intervals are
+95% paired bootstrap intervals.
+
+| | decider-2b v10 | decider-35b-a3b | difference |
+|---|---|---|---|
+| 847 in-task validation rows, accuracy / NLL | 83.2% / 0.444 | 90.0% / 0.329 | +6.7 (+4.5 to +9.0) |
+| OpenJev, 5,252 rows, accuracy / NLL | 63.3% / 0.916 | 68.3% / 0.752 | +5.0 (+3.8 to +6.2) |
+| Mind2Web element and action choice, 1,770 rows | 82.7% / 0.543 | 89.6% / 0.316 | +6.9 (+5.1 to +8.7) |
+| TypeSafe workflow decisions, 102 rows, accuracy / NLL | 80.4% / 0.585 | 86.3% / 0.342 | +5.9 (−2.0 to +13.7) |
+| Bespoke's public suite, 13 subsets, macro / micro | 0.704 / 0.711 | 0.774 / 0.787 | |
+| JevBench public items, easy / standard / hard accuracy | 1.000 / 0.847 / 0.459 | 1.000 / 0.972 / 0.676 | |
+| live MiniWoB++ click tasks, 22 tasks x 8 seeds, greedy play | 90.9% | 97.2% | +6.2 (+1.7 to +10.8) |
+| the same, 6 tasks v10 never used for reward, greedy | 91.7% | 97.9% | +6.2 (0.0 to +14.6) |
+| live MiniWoB++ click tasks, sampled play | 93.2% | 86.4% | −6.8 (−12.5 to −1.7) |
+| the same, 6 held-out tasks, sampled | 91.7% | 79.2% | −12.5 (−25.0 to −2.1) |
+| zero-shot games, win rate, greedy play (234 boards) | 26.5% | 37.2% | +10.7 (+5.6 to +15.8) |
+| zero-shot games, sampled play | 23.7% | 24.1% | +0.4 (−2.4 to +3.3) |
+
+The browser rows show what the RL stage of v10 does and this model lacks: v10's sampled play matches its greedy play because
+RL sharpened the served distribution on those tasks; this model's argmax is right more often, but its distribution still puts
+mass on wrong elements (its sampled play is +3.4 points against v8, which had no RL either). Among the games, the largest
+greedy gains are on the slippery grid (+17 points) and tic-tac-toe (+14); minesweeper stays near zero for every model.
+
+**JevBench public items** (231 items of [Benchmark Heaven](https://benchmarkheaven.com/jev-models); argmax over the exact
+label set with the request the harness's TypeSafe adapter builds). Jev 1.13.0 is at 1.000 / 0.986 / 0.730, djev at 1.000 /
+0.986 / 0.676, SemIf 4B at 1.000 / 0.986 / 0.613 on the same items, from their published per-item outcomes. This model's
+hard-tier misses are on temporal-numeric items (0.33), long policies (0.63) and judge-hard items (0.65); adversarial, trap
+and hard routing items are all correct. Top-label ECE is 0.001 / 0.059 / 0.151 by tier: the model is overconfident on the
+hard tier.
+
+**Bespoke's public suite** (13 human-labelled subsets, 3,880 records in Jev's wire format, answered through `system_one` as
+shipped). Nimble-9B and Jev 1.13.0 numbers are copied from Bespoke's report.
+
+| subset (type) | decider-2b v10 | decider-35b-a3b | Nimble-9B | Jev 1.13.0 |
+|---|---|---|---|---|
+| vitaminc-dev (choice) | 0.639 | 0.795 | 0.766 | 0.801 |
+| massive-en-US (choice; trained) | 0.823 | 0.880 | 0.869 | 0.874 |
+| massive-de-DE (choice, German) | 0.797 | 0.869 | 0.834 | 0.869 |
+| boolq (noul; trained) | 0.803 | 0.887 | 0.860 | 0.897 |
+| squad2 (noul) | 0.776 | 0.749 | 0.806 | 0.829 |
+| paws (noul; trained) | 0.720 | 0.768 | 0.828 | 0.892 |
+| multinli (choice; trained) | 0.856 | 0.910 | 0.853 | 0.829 |
+| civil_comments (noul; trained) | 0.840 | 0.907 | 0.703 | 0.810 |
+| aegis2 (noul) | 0.728 | 0.808 | 0.812 | 0.804 |
+| helpsteer2 (score; trained) | 0.426 | 0.478 | 0.390 | 0.341 |
+| summeval-relevance (score) | 0.354 | 0.483 | 0.492 | 0.350 |
+| summeval-consistency (score) | 0.660 | 0.757 | 0.757 | 0.812 |
+| pubmedqa (choice; trained) | 0.724 | 0.768 | 0.756 | 0.772 |
+| **macro / micro** | 0.704 / 0.711 | **0.774 / 0.787** | 0.748 / 0.759 | 0.760 / 0.773 |
+
+On the six subsets whose training split is not in the mixture the macro accuracy is 0.744. The model is behind Jev where a
+claim has to be checked against evidence that nearly matches it (PAWS, SummEval consistency) and on SQuAD2 answerability.
+
+**Behaviour probes** (teacher-labelled, same probes as the 2B releases): generic-versus-specific bucket choice 1.00 / 1.00,
+catch-all when nothing fits 0.95; command-risk classification 0.933 with no destructive command called safe; browser-agent
+element and action choice 0.938 / 0.938. Scoring a Score level alone against scoring it with its neighbours changes accuracy
+by at most 2 points on five rating datasets, and the per-level fits sum to between 0.92 and 1.07.
+
+## Speed
+
+One NVIDIA B300, bf16, eager PyTorch (`use_graphs=False`), grouped-GEMM experts. A 92-token support ticket with three typed
+questions, and a 5-token chat message with one question:
+
+| setting | latency | throughput |
+|---|---|---|
+| single ticket request, 3 questions | 47 ms | |
+| batch of 64 tickets, 3 questions each | 368 ms | 174 states/s, 522 decisions/s |
+| batch of 64 short messages, 1 question | 111 ms | 575 decisions/s |
+
+decider-2b serves the same tickets at 4 ms with CUDA graphs and about 1,400 to 2,100 decisions/s; this model is for
+workloads where the accuracy gain is worth 3 to 4 times the cost per decision, and for the NVFP4 build on Blackwell (see the
+`-nvfp4` repository).
+
+## Limitations
+
+* No reinforcement-learning stage: stated beliefs about action outcomes were not trained against exact laws, and the served
+  distribution on live browser tasks is less sharp than decider-2b v10's (sampled play 86% against 93%).
+* 65 GB of bf16 weights; one 80 GB GPU is the minimum, and the CUDA-graph and FP8 paths of the helper are untested here.
+* Overconfident on the hardest external items (JevBench hard-tier ECE 0.15) and on some held-out classification sets
+  (TREC 0.16, financial sentiment 0.14) although the aggregate ECE is 0.03 / 0.07.
+* English only. Calibration is measured on public datasets and teacher-labelled probes, not on your traffic. Check it on your
+  own labels before using confidence for routing.
+* The routed experts are the base model's: the fine-tuning changed 2.45B of the 34.7B parameters.
+* Everything else in the decider-2b card's limitations (packed questions see each other, long JSON arrays by position, full
+  label sets against sampled options, abstention wording) applies; those shapes were not re-measured at this size.
+
+## Reproduction
+
+Code, data registry, training and evaluation scripts and the per-version history: https://github.com/Mapika/decider
+(`docs/HISTORY.md`, section "decider-35b-a3b"). The training code for the frozen-expert Muon run is in the repository's
+history document; the merged checkpoint is this repository. Staged with `scripts/stage_release.py` and uploaded with
+`scripts/upload_hf.py`.

chat_template.jinja

@@ -0,0 +1,154 @@
+{%- set image_count = namespace(value=0) %}
+{%- set video_count = namespace(value=0) %}
+{%- macro render_content(content, do_vision_count, is_system_content=false) %}
+    {%- if content is string %}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping %}
+        {%- for item in content %}
+            {%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain images.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set image_count.value = image_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Picture ' ~ image_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|image_pad|><|vision_end|>' }}
+            {%- elif 'video' in item or item.type == 'video' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain videos.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set video_count.value = video_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Video ' ~ video_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|video_pad|><|vision_end|>' }}
+            {%- elif 'text' in item %}
+                {{- item.text }}
+            {%- else %}
+                {{- raise_exception('Unexpected item type in content.') }}
+            {%- endif %}
+        {%- endfor %}
+    {%- elif content is none or content is undefined %}
+        {{- '' }}
+    {%- else %}
+        {{- raise_exception('Unexpected content type.') }}
+    {%- endif %}
+{%- endmacro %}
+{%- if not messages %}
+    {{- raise_exception('No messages provided.') }}
+{%- endif %}
+{%- if tools and tools is iterable and tools is not mapping %}
+    {{- '<|im_start|>system\n' }}
+    {{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>" }}
+    {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {%- if content %}
+            {{- '\n\n' + content }}
+        {%- endif %}
+    {%- endif %}
+    {{- '<|im_end|>\n' }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {{- '<|im_start|>system\n' + content + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" %}
+        {%- set content = render_content(message.content, false)|trim %}
+        {%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
+            {%- set ns.multi_step_tool = false %}
+            {%- set ns.last_query_index = index %}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if ns.multi_step_tool %}
+    {{- raise_exception('No user query found in messages.') }}
+{%- endif %}
+{%- for message in messages %}
+    {%- set content = render_content(message.content, true)|trim %}
+    {%- if message.role == "system" %}
+        {%- if not loop.first %}
+            {{- raise_exception('System message must be at the beginning.') }}
+        {%- endif %}
+    {%- elif message.role == "user" %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in content %}
+                {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+                {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {%- set reasoning_content = reasoning_content|trim %}
+        {%- if loop.index0 > ns.last_query_index %}
+            {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
+        {%- else %}
+            {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- endif %}
+        {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if tool_call.function is defined %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {%- if loop.first %}
+                    {%- if content|trim %}
+                        {{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- else %}
+                        {{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- endif %}
+                {%- else %}
+                    {{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                {%- endif %}
+                {%- if tool_call.arguments is defined %}
+                    {%- for args_name, args_value in tool_call.arguments|items %}
+                        {{- '<parameter=' + args_name + '>\n' }}
+                        {%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %}
+                        {{- args_value }}
+                        {{- '\n</parameter>\n' }}
+                    {%- endfor %}
+                {%- endif %}
+                {{- '</function>\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.previtem and loop.previtem.role != "tool" %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if not loop.last and loop.nextitem.role != "tool" %}
+            {{- '<|im_end|>\n' }}
+        {%- elif loop.last %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- else %}
+        {{- raise_exception('Unexpected message role.') }}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if enable_thinking is defined and enable_thinking is false %}
+        {{- '<think>\n\n</think>\n\n' }}
+    {%- else %}
+        {{- '<think>\n' }}
+    {%- endif %}
+{%- endif %}

config.json

@@ -0,0 +1,97 @@
+{
+  "architectures": [
+    "Qwen3_5MoeForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "attn_output_gate": true,
+  "bos_token_id": null,
+  "dtype": "bfloat16",
+  "eos_token_id": 248044,
+  "full_attention_interval": 4,
+  "head_dim": 256,
+  "hidden_act": "silu",
+  "hidden_size": 2048,
+  "initializer_range": 0.02,
+  "layer_types": [
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention"
+  ],
+  "linear_conv_kernel_dim": 4,
+  "linear_key_head_dim": 128,
+  "linear_num_key_heads": 16,
+  "linear_num_value_heads": 32,
+  "linear_value_head_dim": 128,
+  "mamba_ssm_dtype": "float32",
+  "max_position_embeddings": 262144,
+  "mlp_only_layers": [],
+  "model_type": "qwen3_5_moe_text",
+  "moe_intermediate_size": 512,
+  "mtp_num_hidden_layers": 1,
+  "mtp_use_dedicated_embeddings": false,
+  "num_attention_heads": 16,
+  "num_experts": 256,
+  "num_experts_per_tok": 8,
+  "num_hidden_layers": 40,
+  "num_key_value_heads": 2,
+  "output_router_logits": false,
+  "pad_token_id": null,
+  "partial_rotary_factor": 0.25,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "mrope_interleaved": true,
+    "mrope_section": [
+      11,
+      11,
+      10
+    ],
+    "partial_rotary_factor": 0.25,
+    "rope_theta": 10000000,
+    "rope_type": "default"
+  },
+  "router_aux_loss_coef": 0.001,
+  "shared_expert_intermediate_size": 512,
+  "tie_word_embeddings": false,
+  "transformers_version": "5.17.0",
+  "use_cache": true,
+  "vocab_size": 248320,
+  "experts_implementation": "grouped_mm"
+}

decider/engine.py

@@ -0,0 +1,202 @@
+"""Low-latency inference engine: shape-bucketed CUDA graphs over the one-pass decision model.
+
+Right padding + causal layers => pad positions never influence earlier slots, so no attention
+mask is needed and every (B, T) bucket can be captured once and replayed.  The graph outputs
+option-letter logits for all positions [B, T, K]; slots are gathered outside.
+"""
+import time, torch, torch._dynamo, torch.nn.functional as F
+from decider.model import DecisionModel, collate
+from decider.prompt import build, MAX_OPTIONS
+
+T_BUCKETS = [64, 128, 192, 256, 320, 384, 512, 640, 768, 1024, 1280, 1536, 2048]
+B_BUCKETS = [1, 2, 4, 8, 16, 32, 64]
+GRAPH_MAX_T = 2048          # longer inputs (up to the 32k request budget) run eagerly: compute dominates there, and one graph
+LONG_STEP = 1024            # per (B, T) shape would cost a compile + capture for every new length
+
+
+def _bucket(x, buckets):
+    for b in buckets:
+        if x <= b:
+            return b
+    return None
+
+
+def fused_causal_conv1d_fn(hidden_states, weight, bias=None, activation=None, **kwargs):
+    """Depthwise causal conv (kernel k) as k shifted multiply-adds: fuses under torch.compile,
+    unlike the cuDNN grouped conv fallback (which was ~11% of batched GPU time)."""
+    B, C, T = hidden_states.shape; k = weight.shape[-1]
+    x = F.pad(hidden_states.to(weight.dtype), (k - 1, 0))
+    out = x[:, :, k - 1:k - 1 + T] * weight[:, k - 1][None, :, None]
+    for j in range(k - 1):
+        out = out + x[:, :, j:j + T] * weight[:, j][None, :, None]
+    if bias is not None:
+        out = out + bias[None, :, None]
+    if activation == "silu":
+        out = F.silu(out)
+    elif activation is not None:
+        from transformers.activations import ACT2FN
+        out = ACT2FN[activation](out)
+    return out.to(hidden_states.dtype)
+
+
+def patch_conv():
+    from transformers.models.qwen3_5 import modeling_qwen3_5 as mq
+    mq.causal_conv1d_fn = fused_causal_conv1d_fn
+
+
+def read_slots(out, rows, slots, nopts, temperature, n_per_item):
+    """One gather + one softmax + one device-to-host copy for the whole batch (was: three small kernels and a sync per item).
+    out [B, T, K] logits; rows/slots/nopts: flat python lists, one entry per question; n_per_item: questions per item."""
+    dev = out.device; idx = torch.tensor([rows, slots, nopts], dtype=torch.long).to(dev, non_blocking=True)
+    lg = out[idx[0], idx[1]]                                                                  # [N, K]
+    lg = lg.masked_fill(torch.arange(lg.shape[1], device=dev)[None, :] >= idx[2][:, None], float("-inf"))
+    p = torch.softmax(lg / temperature, -1).cpu()
+    return list(torch.split(p, n_per_item))
+
+
+def fill_ids(items_ids, B, T, pad):
+    import numpy as np
+    a = np.full((B, T), pad, dtype=np.int64)
+    for b, x in enumerate(items_ids): a[b, :len(x)] = x
+    return torch.from_numpy(a)
+
+
+class Engine:
+    """compile: torch.compile the forward (needs use_cache=False; ~1.4x batched, fuses elementwise work).
+    fp8: e4m3 weights + per-token activation scaling on the big linears (Hopper tensor cores).
+    conv_patch: fusable depthwise causal conv instead of the cuDNN fallback."""
+    def __init__(self, path, device="cuda", dtype=torch.bfloat16, use_graphs=True, max_ctx_tokens=1536,
+                 compile=True, fp8=False, conv_patch=True):
+        if conv_patch:
+            patch_conv()
+        self.m = DecisionModel(path, dtype=dtype, grad_ckpt=False).to(device).eval()
+        self.tok = self.m.tok; self.dev = device; self.use_graphs = use_graphs; self.max_ctx = max_ctx_tokens
+        self.core, self.W = self.m.lm.model, self.m.lm.lm_head.weight[self.m.letters].detach().clone()
+        self.cfg = dict(compile=compile, fp8=fp8, conv_patch=conv_patch, graphs=use_graphs)
+        if fp8:
+            from decider.fp8 import convert_to_fp8
+            self.cfg["fp8_layers"] = convert_to_fp8(self.core)
+        if compile:
+            torch._dynamo.config.cache_size_limit = 128
+            self._fwd_impl = torch.compile(self._fwd_eager, dynamic=False)
+        else:
+            self._fwd_impl = self._fwd_eager
+        self.graphs = {}                       # (B, T) -> (static_ids, static_out, graph)
+        self.pool = torch.cuda.graph_pool_handle() if use_graphs else None
+        self.stats = dict(graph_captures=0, forwards=0)
+
+    def _fwd_eager(self, ids):
+        h = self.core(input_ids=ids, use_cache=False).last_hidden_state
+        return F.linear(h, self.W).float()                          # [B, T, K]
+
+    @torch.no_grad()
+    def _fwd(self, ids):
+        return self._fwd_impl(ids)
+
+    def _capture(self, B, T):
+        s_ids = torch.full((B, T), self.tok.pad_token_id, dtype=torch.long, device=self.dev)
+        st = torch.cuda.Stream(); st.wait_stream(torch.cuda.current_stream())
+        with torch.cuda.stream(st):
+            for _ in range(3): self._fwd(s_ids)                     # warm-up: compile / triton autotune
+        torch.cuda.current_stream().wait_stream(st)
+        g = torch.cuda.CUDAGraph()
+        with torch.cuda.graph(g, pool=self.pool):
+            s_out = self._fwd(s_ids)
+        self.stats["graph_captures"] += 1
+        return s_ids, s_out, g
+
+    @torch.no_grad()
+    def logits_all(self, ids):
+        """ids: [B, T] long on device (already right-padded to a bucket). Returns [B, T, K] float."""
+        B, T = ids.shape; self.stats["forwards"] += 1
+        if T > GRAPH_MAX_T:
+            self.stats["long_forwards"] = self.stats.get("long_forwards", 0) + 1
+            return self._fwd_eager(ids)
+        if not self.use_graphs:
+            return self._fwd(ids)
+        key = (B, T)
+        if key not in self.graphs:
+            self.graphs[key] = self._capture(B, T)
+        s_ids, s_out, g = self.graphs[key]
+        s_ids.copy_(ids); g.replay()
+        return s_out
+
+    @torch.no_grad()
+    def score_items(self, items, temperature=1.0):
+        """items: list of dicts from prompt.build. Returns list of [n_q, MAX_OPTIONS] prob tensors (cpu)."""
+        Tmax = max(len(it["ids"]) for it in items)
+        T = _bucket(Tmax, T_BUCKETS) or -(-Tmax // LONG_STEP) * LONG_STEP
+        B = (_bucket(len(items), B_BUCKETS) or len(items)) if T <= GRAPH_MAX_T else len(items)
+        ids = fill_ids([it["ids"] for it in items], B, T, self.tok.pad_token_id)
+        out = self.logits_all(ids.to(self.dev, non_blocking=True))
+        return read_slots(out, [b for b, it in enumerate(items) for _ in it["slots"]], [s for it in items for s in it["slots"]],
+                          [n for it in items for n in it["nopts"]], temperature, [len(it["slots"]) for it in items])
+
+    @torch.no_grad()
+    def score_shared(self, items, temperature=1.0, min_prefix=192):
+        """Rows that start with the same tokens (one state, one question per row): run the shared prefix once, fork its
+        cache (attention KV + delta-net conv/recurrent states) to every row, and run only the question suffixes.
+        Same answers as score_items up to kernel round-off; cost ~ state + sum(questions) instead of n * state."""
+        ids = [it["ids"] for it in items]; n = len(ids)
+        lcp = 0; short = min(len(x) for x in ids) - 1
+        while lcp < short and all(x[lcp] == ids[0][lcp] for x in ids): lcp += 1
+        if n < 2 or lcp < min_prefix:
+            return self.score_items(items, temperature)
+        self.stats["shared_prefix_calls"] = self.stats.get("shared_prefix_calls", 0) + 1
+        pre = torch.tensor(ids[0][:lcp], device=self.dev)[None]
+        cache = self.core(input_ids=pre, use_cache=True).past_key_values
+        cache.reorder_cache(torch.zeros(n, dtype=torch.long, device=self.dev))            # fork: every row gets a copy of row 0
+        Ts = max(len(x) for x in ids) - lcp
+        suf = fill_ids([x[lcp:] for x in ids], n, Ts, self.tok.pad_token_id)
+        h = self.core(input_ids=suf.to(self.dev), past_key_values=cache, use_cache=True).last_hidden_state
+        rows = [b for b, it in enumerate(items) for _ in it["slots"]]; sl = [s - lcp for it in items for s in it["slots"]]
+        idx = torch.tensor([rows, sl], device=self.dev)
+        return read_slots(F.linear(h[idx[0], idx[1]], self.W).float()[:, None, :], list(range(len(rows))), [0] * len(rows),
+                          [n for it in items for n in it["nopts"]], temperature, [len(it["slots"]) for it in items])
+
+    def warmup(self, shapes=((1, 128), (1, 256), (1, 384), (1, 512), (8, 256), (8, 512), (32, 256), (32, 512))):
+        t = time.time()
+        for B, T in shapes:
+            self.logits_all(torch.full((B, T), self.tok.pad_token_id, dtype=torch.long, device=self.dev))
+        torch.cuda.synchronize(); return time.time() - t
+
+
+if __name__ == "__main__":
+    import sys, random, numpy as np
+    from decider import data as D
+    from decider.infer import Decider
+    path = sys.argv[1] if len(sys.argv) > 1 else "runs/r3_v2/model"
+    cfg = dict(compile="nocompile" not in sys.argv[2:], fp8="fp8" in sys.argv[2:], conv_patch="noconv" not in sys.argv[2:])
+    _, evals = D.load_cache("data/tasks.pkl")
+    eng = Engine(path, **cfg); print("engine cfg", eng.cfg)
+    rng = random.Random(0)
+    exs = evals["support_tickets"][:64] + evals["clinc_oos"][:64] + evals["race"][:32]
+    items = [build(e, eng.tok, rng, max_ctx_tokens=1536) for e in exs]
+    # correctness vs eager masked forward (DecisionModel.slot_logits)
+    ref = []
+    with torch.no_grad():
+        for i in range(0, len(items), 16):
+            b = collate(items[i:i + 16], eng.tok.pad_token_id)
+            lg = eng.m.slot_logits(b["input_ids"].cuda(), b["attention_mask"].cuda(), b["slot_idx"].cuda(), b["slot_batch"].cuda(), b["nopts"].cuda())
+            ref.append(torch.softmax(lg, -1).cpu())
+    ref = torch.cat(ref)
+    got = torch.cat(eng.score_items(items))
+    print(f"max |p_graph - p_eager| = {(ref - got).abs().max():.4f} over {len(ref)} questions; argmax agreement {(ref.argmax(1) == got.argmax(1)).float().mean():.4f}")
+    print(f"warmup capture of 8 buckets: {eng.warmup():.1f}s; captures so far {eng.stats['graph_captures']}")
+    # latency: single real requests
+    for name, pool in [("support_tickets", exs[:64]), ("clinc_oos", exs[64:128]), ("race", exs[128:])]:
+        its = [build(e, eng.tok, rng) for e in pool]
+        ts = []
+        for it in its[:40]:
+            torch.cuda.synchronize(); t = time.time(); eng.score_items([it]); torch.cuda.synchronize(); ts.append(time.time() - t)
+        ts = np.array(ts[5:]) * 1000
+        print(f"single request {name:16s}: p50 {np.median(ts):5.1f} ms  p90 {np.percentile(ts, 90):5.1f} ms  (avg {np.mean([len(i['ids']) for i in its]):.0f} tok, {len(its[0]['slots'])} q)")
+        for bs in (8, 32):
+            ts = []
+            for i in range(0, min(len(its), bs * 6), bs):
+                chunk = its[i:i + bs]
+                if len(chunk) < bs: break
+                torch.cuda.synchronize(); t = time.time(); eng.score_items(chunk); torch.cuda.synchronize(); ts.append(time.time() - t)
+            ts = np.array(ts[1:]) * 1000
+            print(f"   batch {bs:2d}: p50 {np.median(ts):6.1f} ms -> {bs/np.median(ts)*1000:6.0f} ctx/s, {bs*len(its[0]['slots'])/np.median(ts)*1000:6.0f} decisions/s")
+    print("stats", eng.stats, "graphs", len(eng.graphs), f"mem {torch.cuda.memory_reserved()/1e9:.1f} GB")

decider/fp8.py

@@ -0,0 +1,54 @@
+"""FP8 (e4m3) linear layers for Hopper via torch._scaled_mm.
+Weights: per-output-channel scales, quantised once.  Activations: per-token dynamic scales.
+Under torch.compile the quantisation ops fuse into the surrounding elementwise work."""
+import torch, torch.nn as nn
+
+E4M3_MAX = 448.0
+
+
+def _quant_rowwise(x):
+    s = x.abs().amax(dim=-1, keepdim=True).float().clamp(min=1e-12) / E4M3_MAX
+    return (x.float() / s).clamp(-E4M3_MAX, E4M3_MAX).to(torch.float8_e4m3fn), s
+
+
+class FP8Linear(nn.Module):
+    def __init__(self, lin: nn.Linear):
+        super().__init__()
+        wq, sw = _quant_rowwise(lin.weight.detach())          # [N,K] fp8, [N,1]
+        self.register_buffer("wq", wq.contiguous())               # [N,K]; passed as wq.t() -> [K,N] column-major, as _scaled_mm wants
+        self.register_buffer("sw_t", sw.t().contiguous())        # [1,N]
+        self.bias = None if lin.bias is None else nn.Parameter(lin.bias.detach().clone(), requires_grad=False)
+        self.in_features, self.out_features = lin.in_features, lin.out_features
+        self.out_dtype = lin.weight.dtype
+
+    def forward(self, x):
+        shp = x.shape[:-1]
+        x2 = x.reshape(-1, self.in_features)
+        xq, sx = _quant_rowwise(x2)
+        y = torch._scaled_mm(xq, self.wq.t(), scale_a=sx, scale_b=self.sw_t, bias=self.bias, out_dtype=self.out_dtype)
+        return y.reshape(*shp, self.out_features)
+
+
+def convert_to_fp8(model, skip=("lm_head",), min_dim=1024):
+    """Replace nn.Linear (with in/out >= min_dim) by FP8Linear in place. Returns count."""
+    n = 0
+    for name, mod in list(model.named_modules()):
+        for cname, child in list(mod.named_children()):
+            full = f"{name}.{cname}" if name else cname
+            if isinstance(child, nn.Linear) and not any(s in full for s in skip) and min(child.in_features, child.out_features) >= min_dim:
+                setattr(mod, cname, FP8Linear(child)); n += 1
+    return n
+
+
+if __name__ == "__main__":
+    import time
+    lin = nn.Linear(2048, 6144, bias=False).cuda().to(torch.bfloat16)
+    f8 = FP8Linear(lin)
+    x = torch.randn(8192, 2048, device="cuda", dtype=torch.bfloat16)
+    ref = lin(x); got = f8(x)
+    print("rel err", ((ref.float() - got.float()).abs().mean() / ref.float().abs().mean()).item())
+    for f, name in [(lin, "bf16 linear"), (f8, "fp8 linear (eager)"), (torch.compile(f8), "fp8 linear (compiled)")]:
+        for _ in range(3): f(x)
+        torch.cuda.synchronize(); t = time.time()
+        for _ in range(20): f(x)
+        torch.cuda.synchronize(); print(f"{name:24s} {(time.time()-t)/20*1000:.3f} ms")

decider/infer.py

@@ -0,0 +1,262 @@
+"""Usable inference API: typed decisions with probabilities, all from one forward pass.
+
+    from decider.infer import Decider
+    d = Decider("runs/r2_full/model")
+    out = d.decide("My card was charged twice for the same purchase.",
+                   [{"question": "Which department should handle this?", "options": ["billing", "technical", "sales"]},
+                    {"question": "How urgent is this?", "options": ["low", "medium", "high"]}])
+    # -> [{'choice': 'billing', 'confidence': 0.97, 'probs': {...}}, {...}]
+"""
+import torch
+from decider.model import DecisionModel, collate
+from decider.prompt import build, MAX_OPTIONS
+from dataclasses import dataclass
+
+
+@dataclass
+class Q:
+    text: str; options: list; gold: int = 0
+
+
+@dataclass
+class Example:
+    context: str; qs: list; task: str = "infer"; image: bytes = None
+
+
+NEUTRAL_NONE = "not listed here"
+
+
+def neutralize_options(options):
+    """The training augmentation used the literal 'none of the above', and the model learned that exact string as an
+    abstain signal (it abstains even on clear cases when the string is offered). Any option that reads like it is
+    rewritten to a neutral phrasing for the model and mapped back in the output."""
+    out, back = [], {}
+    for o in options:
+        key = o.strip().lower()
+        if key.startswith("none of the above") or key in ("none of the above", "none", "n/a", "none of these"):
+            out.append(NEUTRAL_NONE); back[NEUTRAL_NONE] = o
+        else:
+            out.append(o)
+    return out, back
+
+
+class CompiledSchema:
+    def __init__(self, d, rqs, h, index): self.d, self.rqs, self.h, self.index = d, rqs, h, index
+
+    def batch(self, states, max_state_tokens=32768):
+        from decider.systemone import render_state, assemble
+        probs = self.d._se.score(self.h, [render_state(s) for s in states], temperature=self.d.T_schema, max_ctx_tokens=max_state_tokens)
+        return [{"model": self.d.name, "answers": assemble(self.rqs, self.index, [p.tolist() for p in pr])} for pr in probs]
+
+    def __call__(self, state, max_state_tokens=32768):
+        return self.batch([state], max_state_tokens)[0]
+
+
+class Decider:
+    """use_graphs=True (default on CUDA) routes scoring through decider.engine.Engine: shape-bucketed
+    CUDA graphs, ~7x lower single-request latency than eager. Set False for CPU or debugging."""
+    def __init__(self, path, device="cuda", dtype=torch.bfloat16, temperature=None, abstain_below=0.0, use_graphs=None):
+        import json, os
+        cfg = {}
+        try:                                          # model folder may carry decider_config.json (temperature, flags)
+            from huggingface_hub import hf_hub_download
+            cfg_path = os.path.join(path, "decider_config.json") if os.path.isdir(path) else hf_hub_download(path, "decider_config.json")
+            cfg = json.load(open(cfg_path))
+        except Exception:
+            pass
+        if temperature is None:
+            temperature = float(cfg.get("temperature", 1.0))
+        self.neutralize_none = bool(cfg.get("neutralize_none", True))   # v4 and earlier learned the literal string as an abstain signal
+        if use_graphs is None:
+            use_graphs = str(device).startswith("cuda")
+        if use_graphs:
+            from decider.engine import Engine
+            self.eng = Engine(path, device=device, dtype=dtype); self.m = self.eng.m
+        else:
+            self.eng = None; self.m = DecisionModel(path, dtype=dtype, grad_ckpt=False).to(device).eval()
+        self.dev = device; self.T = temperature; self.abstain_below = abstain_below
+        self.name = "decider-" + str(cfg.get("version", "dev"))
+        self.schema_first = bool(cfg.get("schema_first", False)) and self.eng is not None      # default layout. Questions-first (the cacheable one) costs accuracy
+        self.T_schema = float(cfg.get("temperature_schema_first", temperature))                # (about 1.5 points on fixed label sets, more elsewhere): opt in with schema()
+        self.isolated_levels = bool(cfg.get("isolated_levels", False))      # Score levels judged one per row (v8+)
+        self._se = None; self._schemas = {}
+
+    @torch.no_grad()
+    def decide_batch(self, requests, max_ctx_tokens=1536):
+        """requests: list of (context:str, questions:list[dict(question, options)]). One forward pass for everything."""
+        exs, meta = [], []
+        if self.neutralize_none:
+            requests = [(context, [dict(q, options=neutralize_options(q["options"])[0], _back=neutralize_options(q["options"])[1]) for q in qs]) for context, qs in requests]
+        for context, qs in requests:
+            for q in qs:
+                assert 2 <= len(q["options"]) <= MAX_OPTIONS, f"2..{MAX_OPTIONS} options required"
+            exs.append(Example(context, [Q(q["question"], list(q["options"]), 0) for q in qs], "infer"))
+        class _NoShuffle:                      # keep option order as given
+            def shuffle(self, x): pass
+            def sample(self, xs, k): return xs[:k]
+        items = [build(e, self.m.tok, _NoShuffle(), max_options=MAX_OPTIONS, max_ctx_tokens=max_ctx_tokens) for e in exs]
+        if self.eng is not None:
+            probs = torch.cat(self.eng.score_items(items, temperature=self.T))
+        else:
+            b = collate(items, self.m.tok.pad_token_id)
+            logits = self.m.slot_logits(b["input_ids"].to(self.dev), b["attention_mask"].to(self.dev), b["slot_idx"].to(self.dev),
+                                        b["slot_batch"].to(self.dev), b["nopts"].to(self.dev))
+            probs = torch.softmax(logits / self.T, -1).cpu()
+        out, k = [], 0
+        for context, qs in requests:
+            res = []
+            for q in qs:
+                p = probs[k, :len(q["options"])].tolist(); k += 1
+                j = max(range(len(p)), key=p.__getitem__); back = q.get("_back", {})
+                names = [back.get(o, o) for o in q["options"]]
+                res.append(dict(choice=names[j] if p[j] >= self.abstain_below else None, confidence=p[j],
+                                probs={o: pi for o, pi in zip(names, p)}, probs_list=p))
+            out.append(res)
+        return out
+
+    def decide(self, context, questions, **kw):
+        return self.decide_batch([(context, questions)], **kw)[0]
+
+    # ---- Jev-shaped interface (decider.systemone): state + {id: Choice | Score | Noul with criteria}
+    # ---- schema cache (v7+): the questions are run once, requests only run the state (decider.schema_engine)
+    def schema(self, questions, independent=True, isolated=None, compile=False):
+        """Compile a fixed set of Jev-shaped questions: schema(state) -> answers; schema.batch([state, ...]) -> [answers]."""
+        import json
+        from decider.schema_engine import SchemaEngine
+        from decider.systemone import render_question
+        isolated = self.isolated_levels if isolated is None else isolated
+        key = (json.dumps(questions, sort_keys=True, ensure_ascii=False), independent, isolated)
+        if key not in self._schemas:
+            if self._se is None: self._se = SchemaEngine(self.eng)
+            if len(self._schemas) >= 64:                                  # drop the oldest schema and its graphs
+                old = next(iter(self._schemas)); hid = self._schemas.pop(old)[1].id
+                for k in [k for k in self._se.graphs if k[0] == hid]: del self._se.graphs[k]
+            from decider.systemone import plan_rows
+            rqs = {k: render_question(v) for k, v in questions.items()}
+            rows, index = plan_rows(rqs, isolated and independent)
+            h = self._se.prepare(rows, independent=independent, compile=compile)      # compile=True: ~25 s per (batch, length) shape, 1.6x faster after
+            self._schemas[key] = (rqs, h, index)
+        return CompiledSchema(self, *self._schemas[key])
+
+    def system_one(self, state, questions, independent=True, max_state_tokens=32768, max_fwd_tokens=65536, layout=None, isolated=None):
+        layout = layout or ("schema_first" if self.schema_first else "state_first")
+        isolated = (self.isolated_levels if isolated is None else isolated) and independent
+        if layout == "schema_first" and self.eng is not None:
+            return self.schema(questions, independent, isolated)(state, max_state_tokens)
+        """independent=True scores every question in its own row (state + that question only), so adding, removing or
+        reordering questions cannot change any other answer; the state is run once and its cache forked to every
+        question (Engine.score_shared).  independent=False packs all questions behind one copy of the state in one row
+        (later questions can then see earlier question texts)."""
+        from decider.systemone import render_state, render_question, unique_tokens, plan_rows, assemble
+        ctx = render_state(state); rqs = {k: render_question(v) for k, v in questions.items()}
+        opts = (lambda r: neutralize_options(r["options"])[0]) if self.neutralize_none else (lambda r: list(r["options"]))
+        flat, index = plan_rows(rqs, isolated)
+        rows = [[r] for r in flat] if independent else [flat]
+        class _Keep:
+            def shuffle(self, x): pass
+            def sample(self, xs, k): return xs[:k]
+        items = [build(Example(ctx, [Q(r["question"], opts(r), 0) for r in row]), self.m.tok, _Keep(), max_options=MAX_OPTIONS,
+                       max_ctx_tokens=max_state_tokens, layout=layout) for row in rows]
+        with torch.no_grad():
+            if self.eng is not None and len(items) > 1 and layout == "state_first":
+                probs = self.eng.score_shared(items, temperature=self.T)
+            else:
+                probs = []; per = max(1, max_fwd_tokens // max(len(it["ids"]) for it in items))
+                for i in range(0, len(items), per):
+                    if self.eng is not None:
+                        probs += self.eng.score_items(items[i:i + per], temperature=self.T)
+                    else:
+                        bt = collate(items[i:i + per], self.m.tok.pad_token_id)
+                        lg = self.m.slot_logits(*[bt[k].to(self.dev) for k in ("input_ids", "attention_mask", "slot_idx", "slot_batch", "nopts")])
+                        pr = torch.softmax(lg / self.T, -1).cpu(); c = 0
+                        for it in items[i:i + per]:
+                            probs.append(pr[c:c + len(it["slots"])]); c += len(it["slots"])
+        flatp = [p.tolist() for ps in probs for p in ps]
+        return {"model": self.name, "answers": assemble(rqs, index, flatp),
+                "usage": {"input_tokens": unique_tokens(items), "output_tokens": 0}}
+
+    # ---- typed schema interface: {question: {"type": "bool"} | {"type": "choice", "options": [...]}
+    #                                         | {"type": "scale", "legend": {"0": "none", "1": "low", ...}}}
+    @staticmethod
+    def _schema_to_questions(schema):
+        qs = []
+        for qtext, spec in schema.items():
+            t = spec.get("type", "choice")
+            if t == "bool":
+                qs.append(dict(question=qtext, options=["no", "yes"]))
+            elif t == "choice":
+                qs.append(dict(question=qtext, options=list(spec["options"])))
+            elif t == "scale":
+                leg = spec["legend"]
+                keys = sorted(leg, key=lambda k: float(k)) if isinstance(leg, dict) else list(range(len(leg)))
+                labels = [f"{k}: {leg[k]}" if isinstance(leg, dict) else f"{i}: {leg[i]}" for i, k in enumerate(keys)]
+                qs.append(dict(question=qtext, options=labels, _keys=keys, _legend=leg))
+            else:
+                raise ValueError(f"unknown field type {t}")
+        return qs
+
+    def decide_json_batch(self, requests, **kw):
+        """requests: list of (context, schema). Returns one dict per context keyed by question."""
+        qss = [self._schema_to_questions(schema) for _, schema in requests]
+        raw = self.decide_batch([(ctx, qs) for (ctx, _), qs in zip(requests, qss)], **kw)
+        out = []
+        for (ctx, schema), qs, res in zip(requests, qss, raw):
+            o = {}
+            for (qtext, spec), q, r in zip(schema.items(), qs, res):
+                t = spec.get("type", "choice")
+                if t == "bool":
+                    o[qtext] = {"noul": round(r["probs"]["yes"], 4), "type": "noul"}
+                elif t == "choice":
+                    o[qtext] = {"choice": r["choice"], "confidence": round(r["confidence"], 4), "type": "choice",
+                                "probabilities": {k: round(v, 4) for k, v in r["probs"].items()}}
+                else:
+                    p = [r["probs"][lab] for lab in q["options"]]
+                    keys = q["_keys"]; n = len(p)
+                    score = sum(float(k) * pi for k, pi in zip(keys, p))          # expected level on the legend scale
+                    j = max(range(n), key=p.__getitem__)
+                    o[qtext] = {"score": round(score, 2), "confidence": round(p[j], 4), "type": "scale", "legend": q["_legend"],
+                                "probabilities": {str(keys[i]): round(pi, 4) for i, pi in enumerate(p)}}
+            out.append(o)
+        return out
+
+    def decide_json(self, context, schema, **kw):
+        return self.decide_json_batch([(context, schema)], **kw)[0]
+
+
+if __name__ == "__main__":
+    import sys, json, time
+    d = Decider(sys.argv[1] if len(sys.argv) > 1 else "runs/r1_200k/model")
+    demo = [
+        ("My card was charged twice for the same purchase and I want the extra charge refunded.",
+         [{"question": "Which department should handle this?", "options": ["billing", "technical support", "sales"]},
+          {"question": "What is the customer's sentiment?", "options": ["angry", "neutral", "happy"]},
+          {"question": "Does this need a refund action?", "options": ["no", "yes"]}]),
+        ("hey can u turn the lights off in the kitchen",
+         [{"question": "What is the intent?", "options": ["smart home control", "set alarm", "play music", "none of the above"]},
+          {"question": "Is this request toxic?", "options": ["no", "yes"]}]),
+        ("The quarterly report shows revenue fell 12% while costs rose sharply.",
+         [{"question": "What is the financial sentiment?", "options": ["bearish", "neutral", "bullish"]}]),
+    ]
+    t = time.time(); res = d.decide_batch(demo); dt = time.time() - t
+    for (ctx, qs), r in zip(demo, res):
+        print("\n>>", ctx)
+        for q, a in zip(qs, r):
+            print(f"   {q['question']:45s} -> {a['choice']!s:22s} p={a['confidence']:.2f}  " + " ".join(f"{o}:{p:.2f}" for o, p in a['probs'].items()))
+    print(f"\n{sum(len(q) for _, q in demo)} decisions in {dt*1000:.0f} ms (one forward pass)")
+    schema = {
+        "Revenue currently impacted?": {"type": "bool"},
+        "What business impact?": {"type": "choice", "options": ["none", "degraded", "outage"]},
+        "Integration issue present?": {"type": "bool"},
+        "Account health status?": {"type": "choice", "options": ["healthy", "watch", "at risk"]},
+        "Which incident scope?": {"type": "choice", "options": ["single_account", "multi_account", "platform_wide"]},
+        "Security concern present?": {"type": "bool"},
+        "Duplicate charge reported?": {"type": "bool"},
+        "Churn likelihood level?": {"type": "scale", "legend": {"0": "none", "1": "low", "2": "medium", "3": "high"}},
+        "Human attention needed?": {"type": "bool"},
+        "Immediate feature request?": {"type": "bool"},
+    }
+    ctx = ("Hi, since this morning our Stripe webhook integration stopped firing and our checkout is down for all customers. "
+           "We are losing orders every minute and our partner launch is on Thursday. Also I think we got billed twice last week. "
+           "If this is not fixed today we will have to look at other providers.")
+    t = time.time(); js = d.decide_json(ctx, schema); dt = time.time() - t
+    print(f"\n>> {ctx[:80]}...\n" + json.dumps(js, indent=1)[:3000]); print(f"{len(schema)} typed fields in {dt*1000:.0f} ms (one forward pass)")

decider/metrics.py

@@ -0,0 +1,37 @@
+import numpy as np
+
+
+def ece(conf, correct, bins=15):
+    conf = np.asarray(conf); correct = np.asarray(correct, dtype=float)
+    edges = np.linspace(0, 1, bins + 1); e = 0.0
+    for lo, hi in zip(edges[:-1], edges[1:]):
+        m = (conf > lo) & (conf <= hi)
+        if m.any():
+            e += m.mean() * abs(conf[m].mean() - correct[m].mean())
+    return float(e)
+
+
+def aurc(conf, correct):
+    """Area under risk-coverage curve (lower is better)."""
+    order = np.argsort(-np.asarray(conf)); c = np.asarray(correct, dtype=float)[order]
+    risk = np.cumsum(1 - c) / np.arange(1, len(c) + 1)
+    return float(risk.mean())
+
+
+def sel_acc(conf, correct, coverage):
+    order = np.argsort(-np.asarray(conf)); c = np.asarray(correct, dtype=float)[order]
+    n = max(1, int(round(coverage * len(c))))
+    return float(c[:n].mean())
+
+
+def summarize(probs, golds, nopts):
+    """probs [N,K] (masked entries 0), golds [N], nopts [N]."""
+    probs = np.asarray(probs); golds = np.asarray(golds); nopts = np.asarray(nopts)
+    pred = probs.argmax(1); conf = probs.max(1); correct = (pred == golds)
+    p_gold = probs[np.arange(len(golds)), golds]
+    nll = -np.log(np.clip(p_gold, 1e-12, 1)).mean()
+    onehot = np.zeros_like(probs); onehot[np.arange(len(golds)), golds] = 1
+    brier = ((probs - onehot) ** 2).sum(1).mean()
+    return dict(n=int(len(golds)), acc=float(correct.mean()), nll=float(nll), brier=float(brier), ece=ece(conf, correct),
+                aurc=aurc(conf, correct), acc_at_80=sel_acc(conf, correct, 0.8), acc_at_50=sel_acc(conf, correct, 0.5),
+                chance=float((1.0 / nopts).mean()), mean_conf=float(conf.mean()))

decider/model.py

@@ -0,0 +1,47 @@
+"""Backbone -> slot hidden states -> restricted logits over option letters."""
+import torch, torch.nn as nn, torch.nn.functional as F
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from decider.prompt import letter_ids, MAX_OPTIONS
+
+
+class DecisionModel(nn.Module):
+    def __init__(self, name, dtype=torch.bfloat16, grad_ckpt=True):
+        super().__init__()
+        self.tok = AutoTokenizer.from_pretrained(name)
+        self.lm = AutoModelForCausalLM.from_pretrained(name, dtype=dtype)
+        if grad_ckpt:
+            self.lm.gradient_checkpointing_enable()
+        self.register_buffer("letters", torch.tensor(letter_ids(self.tok)), persistent=False)
+
+    def slot_logits(self, input_ids, attention_mask, slot_idx, slot_batch, nopts):
+        """input_ids [B,T]; slot_idx/slot_batch [N] flat slot positions; nopts [N].
+        Returns [N, MAX_OPTIONS] logits with invalid options masked to -inf."""
+        h = self.lm.model(input_ids=input_ids, attention_mask=attention_mask).last_hidden_state
+        hs = h[slot_batch, slot_idx]                                   # [N,H]
+        W = self.lm.lm_head.weight[self.letters]                       # [K,H]
+        logits = F.linear(hs, W).float()                               # [N,K]
+        ar = torch.arange(MAX_OPTIONS, device=logits.device)[None, :]
+        logits = logits.masked_fill(ar >= nopts[:, None], float("-inf"))
+        return logits
+
+    def forward(self, batch):
+        return self.slot_logits(batch["input_ids"], batch["attention_mask"], batch["slot_idx"], batch["slot_batch"], batch["nopts"])
+
+
+def collate(items, pad_id):
+    """items: list of dicts from prompt.build (+ 'task', 'ex_id'). Right-pad."""
+    T = max(len(it["ids"]) for it in items)
+    T = ((T + 63) // 64) * 64          # few distinct shapes -> fewer kernel (re)compiles
+    B = len(items)
+    input_ids = torch.full((B, T), pad_id, dtype=torch.long)
+    attn = torch.zeros((B, T), dtype=torch.long)
+    slot_idx, slot_batch, golds, nopts, tasks, qidx = [], [], [], [], [], []
+    for b, it in enumerate(items):
+        n = len(it["ids"])
+        input_ids[b, :n] = torch.tensor(it["ids"])
+        attn[b, :n] = 1
+        for k, s in enumerate(it["slots"]):
+            slot_idx.append(s); slot_batch.append(b); golds.append(it["golds"][k]); nopts.append(it["nopts"][k])
+            tasks.append(it.get("task", "")); qidx.append(k)
+    return dict(input_ids=input_ids, attention_mask=attn, slot_idx=torch.tensor(slot_idx), slot_batch=torch.tensor(slot_batch),
+                golds=torch.tensor(golds), nopts=torch.tensor(nopts), tasks=tasks, qidx=qidx)

decider/prompt.py

@@ -0,0 +1,157 @@
+"""Prompt construction.  One context, N typed questions, N answer slots.
+
+All N decisions are read from a single forward pass: the logits at each
+"Answer k: (" slot are restricted to the option-letter tokens.  No answer
+letters are ever inserted, so slot k sees the context and all questions but
+no earlier answers (the decisions are conditionally independent given input).
+"""
+import random
+
+LETTERS = "ABCDEFGHIJ"
+NARROW = len(LETTERS)      # <= NARROW options: the original "(A) .. (J)" rendering, tokenized as a string (unchanged since v1)
+MAX_OPTIONS = 255          # width of the label head. > NARROW options: "wide" rendering, one label token per option:
+                           # A..Z then the first 229 two-letter upper-case strings that are single tokens (AA, AB, ...)
+ABSTAIN_PREFIXES = ("none of the above", "none of these", "not listed", "no suitable", "does not apply", "cannot tell")
+ABSTAIN_EXACT = ("other", "unsure", "something else", "neither of these", "other / not covered")
+
+
+def is_abstain_option(o):
+    o = o.strip().lower()
+    return o.startswith(ABSTAIN_PREFIXES) or o in ABSTAIN_EXACT
+
+
+_LABELS = {}
+_OPT_CACHE = {}
+
+
+def _enc_opt(tok, text):
+    """Token ids of ") <option text>" (cached: fixed label sets repeat the same strings millions of times)."""
+    key = (id(tok), text)
+    v = _OPT_CACHE.get(key)
+    if v is None:
+        v = tok.encode(f") {text}", add_special_tokens=False)
+        if len(_OPT_CACHE) < 2_000_000:
+            _OPT_CACHE[key] = v
+    return v
+
+
+def label_table(tok):
+    """(label strings, label token ids), MAX_OPTIONS entries; the first NARROW are A..J so narrow questions are unchanged."""
+    key = id(tok)
+    if key not in _LABELS:
+        import string
+        U = string.ascii_uppercase
+        names = list(U) + [a + b for a in U for b in U]
+        out = []
+        for n in names:
+            t = tok.encode(n, add_special_tokens=False)
+            if len(t) == 1:
+                out.append((n, t[0]))
+            if len(out) == MAX_OPTIONS:
+                break
+        assert len(out) == MAX_OPTIONS and len({i for _, i in out}) == MAX_OPTIONS
+        _LABELS[key] = ([n for n, _ in out], [i for _, i in out],
+                        tok.encode("\n(", add_special_tokens=False))
+    return _LABELS[key]
+
+
+def _select(q, rng, max_options):
+    opts = list(range(len(q.options)))
+    if len(opts) > max_options:
+        # always keep the gold and any abstain-style option (its mere presence must not carry information)
+        forced = {q.gold} | {i for i, o in enumerate(q.options) if is_abstain_option(o)}
+        others = [i for i in opts if i not in forced]
+        opts = rng.sample(others, max_options - len(forced)) + list(forced)
+    rng.shuffle(opts)
+    return opts
+
+
+def _options_ids(tok, q, opts):
+    if len(opts) <= NARROW:
+        return tok.encode("".join(f"\n({LETTERS[j]}) {q.options[oi]}" for j, oi in enumerate(opts)), add_special_tokens=False)
+    _, lab_ids, open_ids = label_table(tok); out = []
+    for j, oi in enumerate(opts):
+        out += open_ids + [lab_ids[j]] + _enc_opt(tok, q.options[oi])
+    return out
+
+
+def build_schema_first(example, tok, rng=None, max_options=NARROW, max_ctx_tokens=1536):
+    """Schema-first layout: all question/option blocks, then the context, then one answer slot per question.
+
+        Question 1: ...\nOptions:\n(A) ...          <- prefix: depends only on the questions, so its cache (attention KV and
+        \n\nQuestion 2: ...                            delta-net states) is computed once per schema and reused for every state
+        \n\nContext:\n<state>\n\nAnswer 1: (\nAnswer 2: (
+
+    The three parts are tokenized separately, so `ids[:prefix_len]` is identical for every state."""
+    rng = rng or random
+    perms = [_select(q, rng, max_options) for q in example.qs]
+    pre = schema_prefix_ids(tok, example.qs, perms)
+    suf, slots = schema_suffix_ids(tok, example.context, len(example.qs), max_ctx_tokens)
+    return dict(ids=pre + suf, slots=[len(pre) + s for s in slots], golds=[p.index(q.gold) if q.gold in p else -1 for p, q in zip(perms, example.qs)],
+                nopts=[len(p) for p in perms], perms=perms, prefix_len=len(pre))
+
+
+def schema_prefix_ids(tok, qs, perms=None):
+    """Token ids of the question/option blocks (the cacheable part of the schema-first layout)."""
+    multi = len(qs) > 1; pre = []
+    for k, q in enumerate(qs):
+        opts = perms[k] if perms is not None else list(range(len(q.options)))
+        pre += tok.encode(f"{chr(10) * 2 if k else ''}Question{' ' + str(k + 1) if multi else ''}: {q.text}\nOptions:", add_special_tokens=False) + _options_ids(tok, q, opts)
+    return pre
+
+
+def schema_suffix_ids(tok, context, n_q, max_ctx_tokens=1536):
+    """Token ids after the schema prefix: the context and one answer slot per question. Returns (ids, slot positions in ids)."""
+    ids = tok.encode("\n\nContext:\n", add_special_tokens=False) + tok.encode(context, add_special_tokens=False)[:max_ctx_tokens]; slots = []
+    for k in range(n_q):
+        ids += tok.encode(f"{chr(10) * 2 if k == 0 else chr(10)}Answer{' ' + str(k + 1) if n_q > 1 else ''}: (", add_special_tokens=False); slots.append(len(ids) - 1)
+    return ids, slots
+
+
+def build(example, tok, rng=None, max_options=NARROW, max_ctx_tokens=1536, layout="state_first"):
+    """Returns dict(ids=list[int], slots=list[int], golds=list[int], nopts=list[int], perms=list[list[int]])."""
+    if layout == "schema_first":
+        return build_schema_first(example, tok, rng, max_options, max_ctx_tokens)
+    rng = rng or random
+    ctx_ids = tok.encode("Context:\n" + example.context, add_special_tokens=False)[:max_ctx_tokens]
+    ids = list(ctx_ids)
+    slots, golds, nopts, perms = [], [], [], []
+    multi = len(example.qs) > 1
+    for k, q in enumerate(example.qs):
+        opts = list(range(len(q.options)))
+        if len(opts) > max_options:
+            # always keep the gold and any abstain-style option (its mere presence must not carry information)
+            forced = {q.gold} | {i for i, o in enumerate(q.options) if is_abstain_option(o)}
+            others = [i for i in opts if i not in forced]
+            keep = rng.sample(others, max_options - len(forced)) + list(forced)
+            opts = keep
+        rng.shuffle(opts)
+        head = f"\n\nQuestion{' ' + str(k + 1) if multi else ''}: {q.text}\nOptions:"
+        tail = f"\nAnswer{' ' + str(k + 1) if multi else ''}: ("
+        if len(opts) <= NARROW:
+            lines = [head] + [f"\n({LETTERS[j]}) {q.options[oi]}" for j, oi in enumerate(opts)] + [tail]
+            piece = tok.encode("".join(lines), add_special_tokens=False)
+        else:                                   # wide: "\n(" + <label token> + ") text", built from ids so every label is one token
+            _, lab_ids, open_ids = label_table(tok)
+            piece = tok.encode(head, add_special_tokens=False)
+            for j, oi in enumerate(opts):
+                piece += open_ids + [lab_ids[j]] + _enc_opt(tok, q.options[oi])
+            piece += tok.encode(tail, add_special_tokens=False)
+        ids.extend(piece)
+        slots.append(len(ids) - 1)          # position of " (" token
+        golds.append(opts.index(q.gold) if q.gold in opts else -1)
+        nopts.append(len(opts))
+        perms.append(opts)
+    return dict(ids=ids, slots=slots, golds=golds, nopts=nopts, perms=perms)
+
+
+def letter_ids(tok):
+    ids = label_table(tok)[1]
+    for j, L in enumerate(LETTERS):
+        assert tok.encode(L, add_special_tokens=False) == [ids[j]], L
+    return ids
+
+
+def render(example, tok, **kw):
+    b = build(example, tok, **kw)
+    return tok.decode(b["ids"])

decider/schema_engine.py

@@ -0,0 +1,143 @@
+"""Schema cache: compute a question schema once, then score states against it.
+
+In production the questions are fixed and only the state changes.  With the schema-first prompt layout
+(prompt.build_schema_first) the question/option blocks are a prefix that does not depend on the state, so their
+cache - attention K/V for the 6 full-attention layers, conv + recurrent state for the 18 delta-net layers - is computed
+once (`prepare`).  A request then runs only "Context: <state>" plus one answer slot per question, as a CUDA graph per
+(batch, length) bucket.  The prefix cache is read-only during a request (nothing is written back), so one copy serves
+every batch and every graph.
+
+    se = SchemaEngine(engine); h = se.prepare([{"question": ..., "options": [...]}, ...])
+    probs = se.score(h, ["state 1", "state 2", ...])          # list of [n_questions, MAX_OPTIONS] tensors
+"""
+import time, types, torch, torch.nn.functional as F
+from decider.prompt import schema_prefix_ids, schema_suffix_ids, MAX_OPTIONS
+from decider.engine import read_slots, fill_ids
+
+TS_BUCKETS = [32, 48, 64, 96, 128, 192, 256, 384, 512, 768, 1024]
+B_BUCKETS = [1, 2, 4, 8, 16, 32, 64]
+
+
+class _Q:
+    def __init__(self, text, options): self.text, self.options = text, options
+
+
+class PrefixCache:
+    """Duck-typed transformers Cache over fixed, read-only prefixes, for one suffix forward pass.
+    A handle holds P prefixes (P = 1: all questions packed in one prefix; P = n_questions: one prefix per question, so every
+    question is scored independently).  A batch of R states has R * P rows; row r * P + p continues prefix p."""
+    def __init__(self, h, R):
+        rep = (lambda t: t.expand(R, *t.shape[1:])) if h.P == 1 else (lambda t: t.repeat(R, *([1] * (t.dim() - 1))))
+        self.tp = h.tpmax; self.k = {i: rep(k) for i, k in h.k.items()}; self.v = {i: rep(v) for i, v in h.v.items()}
+        self.conv = {i: rep(c).contiguous() for i, c in h.conv.items()}
+        self.layers = {i: types.SimpleNamespace(record_past=False, recurrent_states={0: rep(r).contiguous()}) for i, r in h.rec.items()}
+
+    def has_previous_state(self, layer_idx=None, state_idx=None): return True
+    def get_seq_length(self, *a, **k): return self.tp
+    def update(self, key, value, layer_idx, *a, **k): return torch.cat([self.k[layer_idx], key], 2), torch.cat([self.v[layer_idx], value], 2)
+    def update_conv_state(self, x, layer_idx, **k): return torch.cat([self.conv[layer_idx].to(x.dtype), x], -1)
+    def update_recurrent_state(self, s, layer_idx, **k): return s
+
+
+class SchemaEngine:
+    def __init__(self, engine, use_graphs=True):
+        self.e = engine; self.core = engine.core; self.W = engine.W; self.tok = engine.tok; self.dev = engine.dev
+        self.use_graphs = use_graphs and engine.use_graphs; self.graphs = {}; self.stats = dict(prepared=0, captures=0, replays=0, eager=0)
+        self.compile = bool(engine.cfg.get("compile")); self._compiled = {}
+        if self.compile:                        # every compiled schema graph specialises the model frames again (its cache tensors are constants)
+            import torch._dynamo
+            torch._dynamo.config.cache_size_limit = 4096; torch._dynamo.config.accumulated_cache_size_limit = 1 << 16
+
+    @torch.no_grad()
+    def prepare(self, questions, independent=False, compile=False):
+        """questions: [{"question": str, "options": [str]}] in the order answers are wanted.  Runs the prefix(es) once.
+        independent=False: one prefix holding every question (cheapest: a request costs state + n slots).
+        independent=True:  one prefix per question, one row per question (a request costs n * (state + 1 slot); no question
+                           can influence another)."""
+        qs = [_Q(q["question"], list(q["options"])) for q in questions]
+        groups = [[q] for q in qs] if independent else [qs]; pres = [schema_prefix_ids(self.tok, g) for g in groups]
+        h = types.SimpleNamespace(P=len(groups), nq=len(qs), slots_per_row=1 if independent else len(qs), nopts=[len(q.options) for q in qs], tps=[len(p) for p in pres],
+                                  tpmax=max(len(p) for p in pres), k={}, v={}, conv={}, rec={}, id=self.stats["prepared"],
+                                  compile=bool(compile and self.compile))
+        parts = []
+        for pre in pres:
+            out = self.core(input_ids=torch.tensor(pre, device=self.dev)[None], use_cache=True).past_key_values; d = dict(k={}, v={}, conv={}, rec={})
+            for i, layer in enumerate(out.layers):
+                if getattr(layer, "recurrent_states", None) is not None and layer.recurrent_states.get(0) is not None:
+                    d["conv"][i] = layer.conv_states[0]; d["rec"][i] = layer.recurrent_states[0]
+                else:                                                   # right-pad every prefix's K/V to the longest; the mask hides the padding
+                    pad = (0, 0, 0, h.tpmax - len(pre)); d["k"][i] = F.pad(layer.keys, pad); d["v"][i] = F.pad(layer.values, pad)
+            parts.append(d)
+        for name in ("k", "v", "conv", "rec"):
+            getattr(h, name).update({i: torch.cat([d[name][i] for d in parts], 0).clone() for i in parts[0][name]})
+        self.stats["prepared"] += 1
+        return h
+
+    def _fwd(self, ids, cache, mask, pos):
+        hs = self.core(input_ids=ids, past_key_values=cache, attention_mask={"full_attention": mask, "linear_attention": None}, position_ids=pos, use_cache=True).last_hidden_state
+        return F.linear(hs, self.W).float()
+
+    def _static(self, h, R, Ts):
+        """R request slots -> R * P rows.  Mask: a row sees its own prefix (not the padding up to tpmax) and the causal suffix."""
+        ar = torch.arange(Ts, device=self.dev); tps = torch.tensor(h.tps, device=self.dev).repeat(R)                                # [R*P]
+        pre = (torch.arange(h.tpmax, device=self.dev)[None, :] < tps[:, None])[:, None, None, :].expand(-1, 1, Ts, -1)             # [B,1,Ts,tpmax]
+        mask = torch.cat([pre, (ar[:, None] >= ar[None, :])[None, None].expand(len(tps), 1, -1, -1)], 3).contiguous()
+        return PrefixCache(h, R), mask, (tps[:, None] + ar[None, :]).contiguous()
+
+    def _capture(self, h, R, Ts):
+        B = R * h.P
+        ids = torch.full((B, Ts), self.tok.pad_token_id, dtype=torch.long, device=self.dev); cache, mask, pos = self._static(h, R, Ts)
+        fwd = self._fwd
+        if h.compile:                                       # one compiled function per graph (20-30 s each: only for preloaded schemas): the cache tensors are constants of that graph
+            fwd = torch.compile(lambda i: self._fwd(i, cache, mask, pos), dynamic=False)
+            call = lambda: fwd(ids)
+        else:
+            call = lambda: fwd(ids, cache, mask, pos)
+        st = torch.cuda.Stream(); st.wait_stream(torch.cuda.current_stream())
+        with torch.cuda.stream(st):
+            for _ in range(3): call()
+        torch.cuda.current_stream().wait_stream(st)
+        g = torch.cuda.CUDAGraph()
+        with torch.cuda.graph(g, pool=self.e.pool):
+            out = call()
+        self.stats["captures"] += 1
+        return ids, out, g, (cache, mask, pos)
+
+    def warmup(self, h, batch_sizes=(1, 8, 32), state_tokens=(64, 128, 256)):
+        """Capture (and, for a compiled schema, compile) the graphs for these request-batch sizes and suffix lengths ahead of traffic."""
+        t = time.time()
+        for R in batch_sizes:
+            for Ts in state_tokens:
+                Ts = next((x for x in TS_BUCKETS if x >= Ts), TS_BUCKETS[-1])
+                if (h.id, R, Ts) not in self.graphs: self.graphs[(h.id, R, Ts)] = self._capture(h, R, Ts)
+        torch.cuda.synchronize(); return time.time() - t
+
+    def tokenize(self, h, context, max_ctx_tokens=1536):
+        """CPU part of a request (do it outside any GPU lock): -> (suffix ids, slot positions)."""
+        return schema_suffix_ids(self.tok, context, h.slots_per_row, max_ctx_tokens)
+
+    @staticmethod
+    def bucket(n_tokens):
+        return next((t for t in TS_BUCKETS if t >= n_tokens), -(-n_tokens // 256) * 256)
+
+    def score(self, h, contexts, temperature=1.0, max_ctx_tokens=1536):
+        """-> one [n_questions, MAX_OPTIONS] probability tensor per context."""
+        return self.score_rows(h, [self.tokenize(h, c, max_ctx_tokens) for c in contexts], temperature)
+
+    @torch.no_grad()
+    def score_rows(self, h, rows, temperature=1.0):
+        """rows: [(suffix ids, slots)] from tokenize()."""
+        Tmax = max(len(r[0]) for r in rows); Ts = next((t for t in TS_BUCKETS if t >= Tmax), None); n = len(rows)
+        R = next((b for b in B_BUCKETS if b >= n), n) if Ts else n; Ts = Ts or -(-Tmax // 256) * 256
+        ids = fill_ids([x for x, _ in rows for _ in range(h.P)], R * h.P, Ts, self.tok.pad_token_id).to(self.dev, non_blocking=True)
+        if self.use_graphs and Ts <= TS_BUCKETS[-1]:
+            key = (h.id, R, Ts)
+            if key not in self.graphs: self.graphs[key] = self._capture(h, R, Ts)
+            s_ids, s_out, g, _ = self.graphs[key]; s_ids.copy_(ids); g.replay(); out = s_out; self.stats["replays"] += 1
+        else:
+            out = self._fwd(ids, *self._static(h, R, Ts)); self.stats["eager"] += 1
+        if h.P == 1:                                           # packed: n slots in one row per request
+            rws = [r for r in range(n) for _ in range(h.nq)]; sls = [x for _, sl in rows for x in sl]
+        else:                                                  # independent: one slot in each of the request's P rows
+            rws = [r * h.P + p for r in range(n) for p in range(h.P)]; sls = [sl[0] for _, sl in rows for _ in range(h.P)]
+        return read_slots(out, rws, sls, h.nopts * n, temperature, [h.nq] * n)

decider/serve.py

@@ -0,0 +1,289 @@
+"""Micro-batching HTTP server.
+  POST /decide        {"context": str, "schema": {...}}                     -> typed JSON decisions (all questions packed in one row)
+  POST /v1/systemone  {"state": str|object|array, "questions": {id: {...}}} -> the TypeSafe/Jev wire format (decider.systemone):
+                      Choice (up to 255 described options), Score, Noul; every question is scored in its own row, so answers
+                      are independent of each other ("independent": false packs them behind one copy of the state instead).
+Requests arriving within `max_wait_ms` are scored in one forward pass (grouped by length bucket).
+   uvicorn decider.serve:app --host 0.0.0.0 --port 8000     (env: DECIDER_MODEL, DECIDER_MAX_BATCH, DECIDER_MAX_WAIT_MS)
+"""
+import asyncio, os, random, time, threading
+from fastapi import FastAPI, HTTPException
+from pydantic import BaseModel
+from decider.engine import Engine, T_BUCKETS, _bucket
+from decider.prompt import build, MAX_OPTIONS
+from decider.infer import Decider, Example, Q, neutralize_options
+from decider import systemone as S1
+
+MODEL = os.environ.get("DECIDER_MODEL", "runs/r3_v2/model")
+MAX_BATCH = int(os.environ.get("DECIDER_MAX_BATCH", "32"))
+MAX_WAIT_MS = float(os.environ.get("DECIDER_MAX_WAIT_MS", "8"))
+BATCH_WAIT_MS = float(os.environ.get("DECIDER_BATCH_WAIT_MS", "0"))
+MAX_STATE_TOKENS = int(os.environ.get("DECIDER_MAX_STATE_TOKENS", "32768"))
+MAX_FWD_TOKENS = int(os.environ.get("DECIDER_MAX_FWD_TOKENS", "65536"))     # padded tokens per forward pass
+COMPILE = os.environ.get("DECIDER_COMPILE", "1") == "1"
+FP8 = os.environ.get("DECIDER_FP8", "1") == "1"
+app = FastAPI(title="decider")
+MODEL_NAME = "decider"; TEMP = 1.0; TEMP_SCHEMA = 1.0; RELEASE_DATE = "2026-09-17"
+gpu_lock = threading.Lock()            # one GPU job at a time: batched graph replays and shared-prefix requests must not interleave
+SHARED_MIN_TOKENS = int(os.environ.get("DECIDER_SHARED_MIN_TOKENS", "768"))   # independent rows over a state this long share one prefix pass
+eng = None; queue = None; stats = dict(requests=0, batches=0, decisions=0, batch_hist={})
+
+
+class Req(BaseModel):
+    context: str
+    schema_: dict = None
+    model_config = {"populate_by_name": True}
+    def __init__(self, **kw):
+        if "schema" in kw: kw["schema_"] = kw.pop("schema")
+        super().__init__(**kw)
+
+
+class _NoShuffle:
+    def shuffle(self, x): pass
+    def sample(self, xs, k): return xs[:k]
+
+
+def _prepare(context, schema):
+    qs = Decider._schema_to_questions(schema)
+    for q in qs:
+        if getattr(eng, "neutralize_none", True):
+            q["options"], q["_back"] = neutralize_options(q["options"])
+    ex = Example(context, [Q(q["question"], list(q["options"]), 0) for q in qs])
+    it = build(ex, eng.tok, _NoShuffle(), max_options=MAX_OPTIONS, max_ctx_tokens=eng.max_ctx)
+    return qs, it
+
+
+def _format(schema, qs, probs):
+    o = {}
+    for (qtext, spec), q, p in zip(schema.items(), qs, probs):
+        p = p[:len(q["options"])].tolist(); t = spec.get("type", "choice"); j = max(range(len(p)), key=p.__getitem__)
+        back = q.get("_back", {}); names = [back.get(x, x) for x in q["options"]]
+        if t == "bool":
+            o[qtext] = {"noul": round(p[1], 4), "type": "noul"}
+        elif t == "choice":
+            o[qtext] = {"choice": names[j], "confidence": round(p[j], 4), "type": "choice",
+                        "probabilities": {k: round(v, 4) for k, v in zip(names, p)}}
+        else:
+            keys = q["_keys"]; score = sum(float(k) * pi for k, pi in zip(keys, p))
+            o[qtext] = {"score": round(score, 2), "confidence": round(p[j], 4), "type": "scale", "legend": q["_legend"],
+                        "probabilities": {str(keys[i]): round(pi, 4) for i, pi in enumerate(p)}}
+    return o
+
+
+async def _collect(q):
+    """Continuous batching: take what is already queued and go.  While a forward pass runs, new requests pile up and form the
+    next batch, so there is no fixed wait at low load (it cost 1.5 ms per request) and full batches at high load.
+    DECIDER_BATCH_WAIT_MS > 0 restores a short collection window."""
+    batch = [await q.get()]; deadline = time.monotonic() + BATCH_WAIT_MS / 1000
+    while len(batch) < MAX_BATCH:
+        try:
+            batch.append(q.get_nowait())
+        except asyncio.QueueEmpty:
+            timeout = deadline - time.monotonic()
+            if timeout <= 0: break
+            try: batch.append(await asyncio.wait_for(q.get(), timeout))
+            except asyncio.TimeoutError: break
+    return batch
+
+
+async def batcher():
+    loop = asyncio.get_running_loop()
+    while True:
+        batch = await _collect(queue)
+        # sort by length; split into at most two groups when the spread is large (keeps padding small)
+        batch.sort(key=lambda x: len(x[2]["ids"]))
+        groups = [batch]
+        if len(batch) >= 4:
+            lo, hi = len(batch[0][2]["ids"]), len(batch[-1][2]["ids"])
+            if _bucket(hi, T_BUCKETS) != _bucket(lo, T_BUCKETS) and hi > 1.5 * lo:
+                cut = len(batch) // 2; groups = [batch[:cut], batch[cut:]]
+        capped = []                                       # long rows: keep every forward under MAX_FWD_TOKENS padded tokens
+        for g in groups:
+            cur = []
+            for x in g:
+                if cur and (len(cur) + 1) * len(x[2]["ids"]) > MAX_FWD_TOKENS:
+                    capped.append(cur); cur = []
+                cur.append(x)
+            capped.append(cur)
+        for g in capped:
+            items = [it for _, _, it in g]
+            try:
+                probs = await loop.run_in_executor(None, _locked, eng.score_items, items)
+                for (fut, qs, it), p in zip(g, probs):
+                    if not fut.done(): fut.set_result(p)
+            except Exception as e:
+                for fut, _, _ in g:
+                    if not fut.done(): fut.set_exception(e)
+            stats["batches"] += 1; stats["batch_hist"][len(g)] = stats["batch_hist"].get(len(g), 0) + 1
+
+
+@app.on_event("startup")
+async def _start():
+    global eng, queue
+    eng = Engine(MODEL, compile=COMPILE, fp8=FP8, conv_patch=COMPILE); print("[serve] engine", eng.cfg, flush=True)
+    import json
+    global MODEL_NAME, TEMP
+    try: cfg = json.load(open(os.path.join(MODEL, "decider_config.json")))
+    except Exception: cfg = {}
+    eng.neutralize_none = bool(cfg.get("neutralize_none", True)); MODEL_NAME = "decider-" + str(cfg.get("version", "dev"))
+    TEMP = float(os.environ.get("DECIDER_TEMPERATURE", cfg.get("temperature", 1.0)))
+    global RELEASE_DATE; RELEASE_DATE = str(cfg.get("release_date", RELEASE_DATE))
+    global SCHEMA_FIRST, se, squeue, ISOLATED
+    ISOLATED = bool(cfg.get("isolated_levels", False))
+    # the schema cache needs the questions-first layout, which costs accuracy (about 1.5 points on fixed label sets, more on large
+    # label sets and long states): on when the model's config makes it the default, or with DECIDER_SCHEMA_CACHE=1
+    trained = bool(cfg.get("schema_first", False) or cfg.get("schema_first_trained", False))
+    SCHEMA_FIRST = trained and (bool(cfg.get("schema_first", False)) or os.environ.get("DECIDER_SCHEMA_CACHE", "0") == "1")
+    global TEMP_SCHEMA; TEMP_SCHEMA = float(cfg.get("temperature_schema_first", TEMP))
+    if SCHEMA_FIRST:
+        from decider.schema_engine import SchemaEngine
+        se = SchemaEngine(eng); squeue = asyncio.Queue(); asyncio.create_task(schema_batcher()); print("[serve] schema cache on", flush=True)
+        pre = os.environ.get("DECIDER_SCHEMAS")             # JSON file: [{"questions": {...}, "independent": true, "batch_sizes": [1, 8, 32], "state_tokens": [64, 256]}]
+        for spec in (json.load(open(pre)) if pre else []):  # known schemas: prefix computed, graphs compiled and captured before traffic
+            _, h, _ = _schema_handle(spec["questions"], spec.get("independent", True), compile=COMPILE)
+            t = se.warmup(h, spec.get("batch_sizes", (1, 8, 32)), spec.get("state_tokens", (64, 128, 256)))
+            print(f"[serve] preloaded schema with {h.nq} rows, prefix {sum(h.tps)} tokens, graphs ready in {t:.0f}s", flush=True)
+    shapes = [(B, T) for B in (1, 2, 4, 8, 16, 32) for T in T_BUCKETS if T <= eng.max_ctx + 256]
+    if MAX_BATCH > 32: shapes += [(64, T) for T in T_BUCKETS if T <= 512]
+    t = eng.warmup(shapes); print(f"[serve] captured {len(shapes)} graphs in {t:.0f}s", flush=True)
+    queue = asyncio.Queue()
+    asyncio.create_task(batcher())
+
+
+@app.post("/decide")
+async def decide(r: Req):
+    qs, it = await asyncio.get_running_loop().run_in_executor(None, _prepare, r.context, r.schema_)
+    fut = asyncio.get_running_loop().create_future()
+    await queue.put((fut, qs, it))
+    probs = await fut
+    stats["requests"] += 1; stats["decisions"] += len(qs)
+    return _format(r.schema_, qs, probs)
+
+
+SCHEMA_FIRST = False; se = None; squeue = None; schemas = {}            # schema cache (models trained on the questions-first layout, v7+)
+
+
+ISOLATED = False
+
+
+def _schema_handle(questions, independent, compile=False):
+    """Compile (or look up) the question schema: its prefix is run once, requests then only run the state."""
+    import json
+    key = (json.dumps(questions, sort_keys=True, ensure_ascii=False), independent)
+    if key not in schemas:
+        rqs = {k: S1.render_question(v) for k, v in questions.items()}; rows, index = S1.plan_rows(rqs, ISOLATED and independent)
+        with gpu_lock:
+            if len(schemas) >= 128:
+                old = next(iter(schemas)); hid = schemas.pop(old)[1].id
+                for k in [k for k in se.graphs if k[0] == hid]: del se.graphs[k]
+            h = se.prepare(rows, independent=independent, compile=compile)
+        schemas[key] = (rqs, h, index)
+    return schemas[key]
+
+
+seen = {}
+
+
+def _worth_caching(questions, independent):
+    """A schema gets a cached prefix and CUDA graphs from its second request on: one-off schemas go through the generic
+    state-first engine, whose graphs do not depend on the questions, so ad-hoc traffic cannot thrash graph captures."""
+    import json
+    key = (json.dumps(questions, sort_keys=True, ensure_ascii=False), independent)
+    if key in schemas: return True
+    if len(seen) > 50000: seen.clear()
+    seen[key] = seen.get(key, 0) + 1
+    return seen[key] >= int(os.environ.get("DECIDER_SCHEMA_MIN_SEEN", "2"))
+
+
+def _score_schema(h, rows):
+    with gpu_lock:
+        return se.score_rows(h, rows, temperature=TEMP_SCHEMA)
+
+
+async def schema_batcher():
+    """Requests that share a schema and arrive within the window are scored in one forward pass over their states."""
+    loop = asyncio.get_running_loop()
+    while True:
+        batch = await _collect(squeue)
+        groups = {}                                        # one forward per (schema, length bucket): short states are not padded to long ones
+        for fut, h, row in batch: groups.setdefault((h.id, se.bucket(len(row[0]))), (h, []))[1].append((fut, row))
+        for h, items in groups.values():
+            step = max(1, MAX_BATCH // h.P)
+            for i in range(0, len(items), step):
+                chunk = items[i:i + step]
+                try:
+                    probs = await loop.run_in_executor(None, _score_schema, h, [c for _, c in chunk])
+                    for (fut, _), p in zip(chunk, probs):
+                        if not fut.done(): fut.set_result(p)
+                except Exception as e:
+                    for fut, _ in chunk:
+                        if not fut.done(): fut.set_exception(e)
+            stats["schema_batches"] = stats.get("schema_batches", 0) + 1
+
+
+def _locked(fn, items):
+    with gpu_lock:
+        return fn(items, temperature=TEMP)          # fitted temperature from decider_config.json
+
+
+class S1Req(BaseModel):
+    state: object
+    questions: dict
+    model: str | None = None
+    independent: bool = True
+    layout: str | None = None            # "state_first" forces the uncached layout on a schema-first model
+
+
+def _prepare_s1(state, questions, independent):
+    ctx = S1.render_state(state); rqs = {k: S1.render_question(v) for k, v in questions.items()}
+    flat, index = S1.plan_rows(rqs, ISOLATED and independent)
+    rows = [[r] for r in flat] if independent else [flat]
+    items = [build(Example(ctx, [Q(r["question"], list(r["options"]), 0) for r in row]), eng.tok, _NoShuffle(), max_options=MAX_OPTIONS,
+                   max_ctx_tokens=MAX_STATE_TOKENS) for row in rows]
+    return (rqs, index), items
+
+
+@app.post("/v1/systemone")
+async def systemone(r: S1Req):
+    loop = asyncio.get_running_loop()
+    if SCHEMA_FIRST and r.layout != "state_first" and _worth_caching(r.questions, r.independent):
+        try:
+            rqs, h, index = await loop.run_in_executor(None, _schema_handle, r.questions, r.independent)
+        except ValueError as e:
+            raise HTTPException(422, str(e))
+        row = await loop.run_in_executor(None, lambda: se.tokenize(h, S1.render_state(r.state), MAX_STATE_TOKENS))      # CPU work stays off the GPU lock
+        fut = loop.create_future(); await squeue.put((fut, h, row)); p = await fut
+        stats["requests"] += 1; stats["decisions"] += len(rqs); stats["schema_requests"] = stats.get("schema_requests", 0) + 1
+        return {"model": MODEL_NAME, "answers": S1.assemble(rqs, index, [pk.tolist() for pk in p]),
+                "usage": {"input_tokens": len(row[0]) * h.P, "cached_tokens": sum(h.tps), "output_tokens": 0}}
+    try:
+        (rqs, index), items = await loop.run_in_executor(None, _prepare_s1, r.state, r.questions, r.independent)
+    except ValueError as e:
+        raise HTTPException(422, str(e))
+    if len(items) > 1 and min(len(it["ids"]) for it in items) >= SHARED_MIN_TOKENS:
+        res = await loop.run_in_executor(None, _locked, eng.score_shared, items)      # long state: run it once, fork the cache per question
+        stats["shared_prefix_requests"] = stats.get("shared_prefix_requests", 0) + 1
+    else:
+        futs = []
+        for it in items:
+            f = loop.create_future(); futs.append(f); await queue.put((f, None, it))
+        res = await asyncio.gather(*futs)
+    probs = [p for ps in res for p in ps]                                   # one prob row per question, request order
+    stats["requests"] += 1; stats["decisions"] += len(rqs)
+    return {"model": MODEL_NAME, "answers": S1.assemble(rqs, index, [p.tolist() for p in probs]),
+            "usage": {"input_tokens": S1.unique_tokens(items), "output_tokens": 0}}
+
+
+@app.get("/v1/models")
+async def models():
+    return {"models": [{"name": MODEL_NAME, "description": "decider: one-pass typed decisions with calibrated probabilities", "release_date": RELEASE_DATE}]}
+
+
+@app.get("/health")
+async def health():
+    return {"ok": eng is not None, "model": MODEL}
+
+
+@app.get("/stats")
+async def get_stats():
+    return dict(stats, engine=eng.stats if eng else None, graphs=len(eng.graphs) if eng else 0)

decider/systemone.py

@@ -0,0 +1,139 @@
+"""Jev-shaped requests on top of the decider prompt format (same wire format as TypeSafe's POST /v1/systemone).
+
+    state      str | dict | list            JSON state is serialised compactly; questions may name a part by path (`ticket.messages[0].text`)
+    questions  {id: {"type": "choice", "instructions": ..., "criteria": {name: description | {...} | [...] | None}}      up to 255 options
+                     {"type": "score",  "instructions": ..., "criteria": [level 0 description, level 1 description, ...]}  2..10 levels
+                     {"type": "noul",   "instructions": ..., "criteria": {"true": ..., "false": ...} (optional)}}
+    ids are never shown to the model.  `instructions` and every description may be a string or any JSON value.
+"""
+import json, math
+
+MAX_CHOICE, MAX_LEVELS = 255, 10
+
+
+def _txt(v):
+    return v if isinstance(v, str) else json.dumps(v, ensure_ascii=False)
+
+
+ANNOTATE_MIN = 8
+
+
+def annotate_indices(x, min_len=ANNOTATE_MIN):
+    """Write each element's position into long arrays ({"_index": i, ...}).  A path such as `records[47].text` otherwise makes
+    the model count 47 elements; with the index written down it is a lookup (json_k64 probe: 0.49 -> 0.57 accuracy)."""
+    if isinstance(x, list):
+        if len(x) >= min_len:
+            return [({"_index": i, **annotate_indices(v, min_len)} if isinstance(v, dict) else {"_index": i, "value": annotate_indices(v, min_len)}) for i, v in enumerate(x)]
+        return [annotate_indices(v, min_len) for v in x]
+    if isinstance(x, dict):
+        return {k: annotate_indices(v, min_len) for k, v in x.items()}
+    return x
+
+
+def render_state(state, index_arrays=True):
+    if isinstance(state, str):
+        return state
+    return json.dumps(annotate_indices(state) if index_arrays else state, ensure_ascii=False)
+
+
+def render_question(spec):
+    """-> dict(question=str, options=[str], type=..., names=[...])  (names: what the answer reports for each option)"""
+    t = spec.get("type", "choice"); ins = _txt(spec.get("instructions", spec.get("question", ""))); crit = spec.get("criteria", spec.get("options"))
+    if not ins:
+        raise ValueError("question without instructions")
+    if t == "choice":
+        if isinstance(crit, (list, tuple)):
+            crit = {str(c): None for c in crit}
+        if not isinstance(crit, dict) or not 2 <= len(crit) <= MAX_CHOICE:
+            raise ValueError(f"choice criteria: a map of 2..{MAX_CHOICE} options")
+        names = list(crit); opts = [n if crit[n] in (None, "") else f"{n}: {_txt(crit[n])}" for n in names]
+    elif t == "score":
+        if isinstance(crit, dict):                                   # legend form {"0": "...", "1": "..."}
+            crit = [crit[k] for k in sorted(crit, key=float)]
+        if not isinstance(crit, (list, tuple)) or not 2 <= len(crit) <= MAX_LEVELS:
+            raise ValueError(f"score criteria: an ordered list of 2..{MAX_LEVELS} level descriptions")
+        names = list(range(len(crit))); opts = [f"{i}: {_txt(c)}" for i, c in enumerate(crit)]
+    elif t in ("noul", "bool"):
+        names = [False, True]; c = crit or {}
+        f, tr = c.get("false", c.get(False)), c.get("true", c.get(True))
+        opts = ["no" if f in (None, "") else f"no: {_txt(f)}", "yes" if tr in (None, "") else f"yes: {_txt(tr)}"]
+    else:
+        raise ValueError(f"unknown question type {t!r}")
+    return dict(question=ins, options=opts, type="noul" if t == "bool" else t, names=names, legend=[_txt(c) for c in crit] if t == "score" else None,
+                isolated=bool(spec.get("isolated", True)))
+
+
+# ---- isolated levels: every Score level is judged in its own row, without its number or its neighbours
+ISOLATED = "{q}\nProposed answer: {level}\nDoes the proposed answer fit?"
+_NUM = None
+
+
+def strip_level_number(text):
+    """"2: somewhat" -> "somewhat" (dataset legends carry the number; an isolated level must not)."""
+    import re
+    return re.sub(r"^\s*-?\d+\s*:\s*", "", text)
+
+
+def isolated_rows(question, levels):
+    """-> one yes/no question per level: [(question text, ["no", "yes"])]."""
+    return [(ISOLATED.format(q=question, level=strip_level_number(l)), ["no", "yes"]) for l in levels]
+
+
+def combine_isolated(p_yes):
+    """Per-level P(fits), each computed without reference to any other level -> a distribution over levels.
+    Also returns the unnormalised mass: near 1 when exactly one level fits, low when none does, high when several do."""
+    tot = sum(p_yes) or 1e-9
+    return [x / tot for x in p_yes], tot
+
+
+def plan_rows(rqs, isolated=True):
+    """One scoring row per question; a Score question with isolated levels becomes one yes/no row per level.
+    -> (rows [{"question", "options"}], index [(id, "iso" | "list", first row, n rows)])"""
+    rows, index = [], []
+    for k, r in rqs.items():
+        if isolated and r["type"] == "score" and r.get("isolated", True):
+            rws = isolated_rows(r["question"], r["legend"]); index.append((k, "iso", len(rows), len(rws))); rows += [dict(question=t, options=o) for t, o in rws]
+        else:
+            index.append((k, "list", len(rows), 1)); rows.append(dict(question=r["question"], options=r["options"]))
+    return rows, index
+
+
+def assemble(rqs, index, probs):
+    """probs: one probability list per row (plan_rows order) -> {id: answer}."""
+    out = {}
+    for k, kind, s, n in index:
+        if kind == "iso":
+            fit = [float(probs[s + j][1]) for j in range(n)]; p, mass = combine_isolated(fit); a = format_answer(rqs[k], p)
+            a["level_fit"] = {str(j): round(x, 4) for j, x in enumerate(fit)}; a["fit_mass"] = round(mass, 4); out[k] = a
+        else:
+            out[k] = format_answer(rqs[k], probs[s])
+    return out
+
+
+def certainty(p):
+    """1 - normalised entropy: 1 when all mass is on one option, 0 when the distribution is flat."""
+    h = -sum(x * math.log(x) for x in p if x > 0)
+    return max(0.0, 1.0 - h / math.log(len(p))) if len(p) > 1 else 1.0
+
+
+def format_answer(rq, p, nd=4):
+    """rq: render_question output; p: probabilities in option order."""
+    p = [float(x) for x in p[:len(rq["options"])]]; s = sum(p) or 1.0; p = [x / s for x in p]
+    j = max(range(len(p)), key=p.__getitem__)
+    if rq["type"] == "noul":
+        return {"type": "noul", "noul": round(p[1], nd)}
+    if rq["type"] == "choice":
+        return {"type": "choice", "choice": rq["names"][j], "confidence": round(p[j], nd), "certainty": round(certainty(p), nd),
+                "probabilities": {n: round(x, nd) for n, x in zip(rq["names"], p)}}
+    return {"type": "score", "score": round(sum(i * x for i, x in enumerate(p)), 2), "confidence": round(p[j], nd), "certainty": round(certainty(p), nd),
+            "legend": {str(i): d for i, d in enumerate(rq["legend"])}, "probabilities": {str(i): round(x, nd) for i, x in enumerate(p)}}
+
+
+def unique_tokens(items):
+    """Input tokens of a request whose rows share a prefix (the state): the prefix counts once."""
+    ids = [it["ids"] for it in items]
+    if len(ids) < 2:
+        return sum(len(x) for x in ids)
+    lcp = 0; short = min(len(x) for x in ids)
+    while lcp < short and all(x[lcp] == ids[0][lcp] for x in ids): lcp += 1
+    return lcp + sum(len(x) - lcp for x in ids)

decider_config.json

@@ -0,0 +1,13 @@
+{
+ "temperature": 1.08,
+ "neutralize_none": false,
+ "version": "35b-a3b-v1",
+ "base": "Qwen/Qwen3.5-35B-A3B-Base",
+ "max_options": 255,
+ "max_state_tokens": 32768,
+ "schema_first": false,
+ "schema_first_trained": true,
+ "isolated_levels": true,
+ "release_date": "2026-09-20",
+ "stage": "one public-mixture epoch (463M tokens) with Muon, routed experts frozen; runs/moe_optimizer_ab_v1/muon step 16287; temperature fitted on the in-task regression half"
+}

eval_results.json

@@ -0,0 +1,1298 @@
+{
+ "model": "decider-35b-a3b v1",
+ "temperature": 1.075,
+ "protocol": "public regression set rebuilt on this machine: eval half of data/mixture_full.pkl, 95 tasks (67 in-task / 28 held-out), state-first layout, max_ctx 1536, large label sets sub-sampled to 10 options; metrics at the stored temperature",
+ "aggregate": {
+  "in_task": {
+   "acc": 0.8551758035695551,
+   "nll": 0.3571210244530074,
+   "ece": 0.026005523425337858
+  },
+  "heldout": {
+   "acc": 0.8104967349952829,
+   "nll": 0.4970387108931333,
+   "ece": 0.06886512604372556
+  }
+ },
+ "results": {
+  "clinc_oos": {
+   "n": 1500,
+   "heldout": false,
+   "acc": 0.9826666666666667,
+   "nll": 0.06116773560643196,
+   "brier": 0.026913650333881378,
+   "ece": 0.004699912309646655,
+   "aurc": 0.001413522036819675,
+   "acc_at_80": 0.9975,
+   "acc_at_50": 1.0,
+   "chance": 0.1,
+   "mean_conf": 0.9831148982048035
+  },
+  "banking77": {
+   "n": 1500,
+   "heldout": false,
+   "acc": 0.9866666666666667,
+   "nll": 0.051965903490781784,
+   "brier": 0.022627051919698715,
+   "ece": 0.016074508627255804,
+   "aurc": 0.0006460488654478452,
+   "acc_at_80": 0.9991666666666666,
+   "acc_at_50": 1.0,
+   "chance": 0.1,
+   "mean_conf": 0.9768534898757935
+  },
+  "massive_intent": {
+   "n": 1500,
+   "heldout": false,
+   "acc": 0.9633333333333334,
+   "nll": 0.11785730719566345,
+   "brier": 0.0557381696999073,
+   "ece": 0.008312105695406588,
+   "aurc": 0.0031118976074550235,
+   "acc_at_80": 0.9966666666666667,
+   "acc_at_50": 0.9986666666666667,
+   "chance": 0.1,
+   "mean_conf": 0.9660524129867554
+  },
+  "massive_scenario": {
+   "n": 1500,
+   "heldout": true,
+   "acc": 0.7993333333333333,
+   "nll": 0.5332828760147095,
+   "brier": 0.271218866109848,
+   "ece": 0.02829863039652506,
+   "aurc": 0.055478553769578394,
+   "acc_at_80": 0.8808333333333334,
+   "acc_at_50": 0.9733333333333334,
+   "chance": 0.1,
+   "mean_conf": 0.7986482977867126
+  },
+  "bitext_support": {
+   "n": 3000,
+   "heldout": false,
+   "acc": 0.9993333333333333,
+   "nll": 0.0021056807599961758,
+   "brier": 0.0011287310626357794,
+   "ece": 0.0003376175860563954,
+   "aurc": 1.1120011870401867e-06,
+   "acc_at_80": 1.0,
+   "acc_at_50": 1.0,
+   "chance": 0.09999999999999998,
+   "mean_conf": 0.9992170333862305
+  },
+  "support_tickets": {
+   "n": 4500,
+   "heldout": false,
+   "acc": 0.5933333333333334,
+   "nll": 0.9494668245315552,
+   "brier": 0.5020924210548401,
+   "ece": 0.017903674913777236,
+   "aurc": 0.20277115689216718,
+   "acc_at_80": 0.6583333333333333,
+   "acc_at_50": 0.7777777777777778,
+   "chance": 0.22777777777777777,
+   "mean_conf": 0.5856716632843018
+  },
+  "ag_news": {
+   "n": 1500,
+   "heldout": false,
+   "acc": 0.9346666666666666,
+   "nll": 0.20106011629104614,
+   "brier": 0.10288789123296738,
+   "ece": 0.010400939603646572,
+   "aurc": 0.013618430266727854,
+   "acc_at_80": 0.9783333333333334,
+   "acc_at_50": 0.9893333333333333,
+   "chance": 0.25,
+   "mean_conf": 0.9317083358764648
+  },
+  "dbpedia": {
+   "n": 1500,
+   "heldout": false,
+   "acc": 0.9926666666666667,
+   "nll": 0.019056648015975952,
+   "brier": 0.010244570672512054,
+   "ece": 0.00390036821365361,
+   "aurc": 6.964876547762933e-05,
+   "acc_at_80": 1.0,
+   "acc_at_50": 1.0,
+   "chance": 0.1,
+   "mean_conf": 0.9953445792198181
+  },
+  "yahoo_topics": {
+   "n": 1500,
+   "heldout": false,
+   "acc": 0.7726666666666666,
+   "nll": 0.6962489485740662,
+   "brier": 0.3215087354183197,
+   "ece": 0.04092799503604572,
+   "aurc": 0.07648607099774656,
+   "acc_at_80": 0.86,
+   "acc_at_50": 0.944,
+   "chance": 0.1,
+   "mean_conf": 0.7425376176834106
+  },
+  "newsgroups": {
+   "n": 1462,
+   "heldout": false,
+   "acc": 0.8502051983584131,
+   "nll": 0.449008584022522,
+   "brier": 0.21066278219223022,
+   "ece": 0.03997525807199984,
+   "aurc": 0.026273657246117587,
+   "acc_at_80": 0.9435897435897436,
+   "acc_at_50": 0.9931600547195623,
+   "chance": 0.09999999999999999,
+   "mean_conf": 0.88435298204422
+  },
+  "bbc_news": {
+   "n": 1000,
+   "heldout": true,
+   "acc": 0.944,
+   "nll": 0.13256792724132538,
+   "brier": 0.07240951806306839,
+   "ece": 0.027219559907913193,
+   "aurc": 0.004171691057266529,
+   "acc_at_80": 0.995,
+   "acc_at_50": 1.0,
+   "chance": 0.20000000000000004,
+   "mean_conf": 0.9385547637939453
+  },
+  "trec": {
+   "n": 500,
+   "heldout": true,
+   "acc": 0.832,
+   "nll": 0.587404727935791,
+   "brier": 0.28857845067977905,
+   "ece": 0.1598434355854988,
+   "aurc": 0.06672534257121955,
+   "acc_at_80": 0.8925,
+   "acc_at_50": 0.952,
+   "chance": 0.16666666666666663,
+   "mean_conf": 0.6734781265258789
+  },
+  "student_questions": {
+   "n": 1500,
+   "heldout": true,
+   "acc": 0.954,
+   "nll": 0.2031065672636032,
+   "brier": 0.08712532371282578,
+   "ece": 0.08987796316544214,
+   "aurc": 0.006595183463977914,
+   "acc_at_80": 0.9916666666666667,
+   "acc_at_50": 0.9933333333333333,
+   "chance": 0.25,
+   "mean_conf": 0.865302562713623
+  },
+  "dolly_category": {
+   "n": 1500,
+   "heldout": true,
+   "acc": 0.354,
+   "nll": 1.631442666053772,
+   "brier": 0.7210286259651184,
+   "ece": 0.09806547661622367,
+   "aurc": 0.4519247410276184,
+   "acc_at_80": 0.38666666666666666,
+   "acc_at_50": 0.47333333333333333,
+   "chance": 0.125,
+   "mean_conf": 0.4256460666656494
+  },
+  "imdb": {
+   "n": 1500,
+   "heldout": false,
+   "acc": 0.9706666666666667,
+   "nll": 0.0796746090054512,
+   "brier": 0.04415109381079674,
+   "ece": 0.009556351860364288,
+   "aurc": 0.0021247781519913296,
+   "acc_at_80": 0.9966666666666667,
+   "acc_at_50": 1.0,
+   "chance": 0.5,
+   "mean_conf": 0.9738101959228516
+  },
+  "sst2": {
+   "n": 872,
+   "heldout": false,
+   "acc": 0.963302752293578,
+   "nll": 0.1075686514377594,
+   "brier": 0.058069903403520584,
+   "ece": 0.007435061174248355,
+   "aurc": 0.004894615707237302,
+   "acc_at_80": 0.9914040114613181,
+   "acc_at_50": 0.9977064220183486,
+   "chance": 0.5,
+   "mean_conf": 0.9640254974365234
+  },
+  "sst5": {
+   "n": 1500,
+   "heldout": false,
+   "acc": 0.6326666666666667,
+   "nll": 0.8465693593025208,
+   "brier": 0.5005199313163757,
+   "ece": 0.02862738460302352,
+   "aurc": 0.30424582658146127,
+   "acc_at_80": 0.6608333333333334,
+   "acc_at_50": 0.6973333333333334,
+   "chance": 0.2,
+   "mean_conf": 0.6309818625450134
+  },
+  "yelp": {
+   "n": 1500,
+   "heldout": false,
+   "acc": 0.7266666666666667,
+   "nll": 0.6343568563461304,
+   "brier": 0.37920209765434265,
+   "ece": 0.04076567333936691,
+   "aurc": 0.13746089140338316,
+   "acc_at_80": 0.7716666666666666,
+   "acc_at_50": 0.8533333333333334,
+   "chance": 0.2,
+   "mean_conf": 0.739836573600769
+  },
+  "amazon_stars": {
+   "n": 1500,
+   "heldout": false,
+   "acc": 0.6213333333333333,
+   "nll": 0.8520724177360535,
+   "brier": 0.4818972945213318,
+   "ece": 0.03010182730356851,
+   "aurc": 0.21266833331300722,
+   "acc_at_80": 0.6741666666666667,
+   "acc_at_50": 0.7693333333333333,
+   "chance": 0.2,
+   "mean_conf": 0.6432384252548218
+  },
+  "emotion": {
+   "n": 1500,
+   "heldout": false,
+   "acc": 0.9086666666666666,
+   "nll": 0.22119809687137604,
+   "brier": 0.12285951524972916,
+   "ece": 0.015565859278043073,
+   "aurc": 0.012805207153182863,
+   "acc_at_80": 0.9808333333333333,
+   "acc_at_50": 0.9973333333333333,
+   "chance": 0.16666666666666666,
+   "mean_conf": 0.9207584857940674
+  },
+  "go_emotions": {
+   "n": 1500,
+   "heldout": false,
+   "acc": 0.794,
+   "nll": 0.5889915227890015,
+   "brier": 0.28999069333076477,
+   "ece": 0.029260090112686165,
+   "aurc": 0.0672639072365525,
+   "acc_at_80": 0.8775,
+   "acc_at_50": 0.9533333333333334,
+   "chance": 0.1,
+   "mean_conf": 0.7891288995742798
+  },
+  "tweet_sentiment": {
+   "n": 1500,
+   "heldout": false,
+   "acc": 0.7393333333333333,
+   "nll": 0.5619461536407471,
+   "brier": 0.3484695255756378,
+   "ece": 0.053492017388343804,
+   "aurc": 0.12272191307348365,
+   "acc_at_80": 0.8075,
+   "acc_at_50": 0.8733333333333333,
+   "chance": 0.3333333333333333,
+   "mean_conf": 0.706974446773529
+  },
+  "tweet_emotion": {
+   "n": 1421,
+   "heldout": false,
+   "acc": 0.8648838845883181,
+   "nll": 0.3760232925415039,
+   "brier": 0.19416743516921997,
+   "ece": 0.048255335898738254,
+   "aurc": 0.027839586844617643,
+   "acc_at_80": 0.9481090589270009,
+   "acc_at_50": 0.9887323943661972,
+   "chance": 0.25,
+   "mean_conf": 0.8190691471099854
+  },
+  "tweet_irony": {
+   "n": 784,
+   "heldout": true,
+   "acc": 0.860969387755102,
+   "nll": 0.3990391194820404,
+   "brier": 0.24129782617092133,
+   "ece": 0.12924721418898932,
+   "aurc": 0.04448957010495625,
+   "acc_at_80": 0.9202551834130781,
+   "acc_at_50": 0.9642857142857143,
+   "chance": 0.5,
+   "mean_conf": 0.7329258322715759
+  },
+  "fin_sentiment": {
+   "n": 1500,
+   "heldout": true,
+   "acc": 0.8386666666666667,
+   "nll": 0.46592089533805847,
+   "brier": 0.27144962549209595,
+   "ece": 0.13573047741254168,
+   "aurc": 0.06709094165414199,
+   "acc_at_80": 0.8941666666666667,
+   "acc_at_50": 0.9466666666666667,
+   "chance": 0.3333333333333333,
+   "mean_conf": 0.7034342288970947
+  },
+  "cr_reviews": {
+   "n": 753,
+   "heldout": true,
+   "acc": 0.9136786188579017,
+   "nll": 0.1997925043106079,
+   "brier": 0.12444894015789032,
+   "ece": 0.032485922019320164,
+   "aurc": 0.013022305207380291,
+   "acc_at_80": 0.978405315614618,
+   "acc_at_50": 0.9973404255319149,
+   "chance": 0.5,
+   "mean_conf": 0.940420389175415
+  },
+  "counterfactual": {
+   "n": 1500,
+   "heldout": false,
+   "acc": 0.948,
+   "nll": 0.12409600615501404,
+   "brier": 0.07306990027427673,
+   "ece": 0.015566750446955315,
+   "aurc": 0.005129710883211171,
+   "acc_at_80": 0.995,
+   "acc_at_50": 0.9986666666666667,
+   "chance": 0.5,
+   "mean_conf": 0.9607836008071899
+  },
+  "subjectivity": {
+   "n": 1500,
+   "heldout": false,
+   "acc": 0.9773333333333334,
+   "nll": 0.06512857228517532,
+   "brier": 0.03518040478229523,
+   "ece": 0.006610139727592426,
+   "aurc": 0.001447138121234917,
+   "acc_at_80": 0.9983333333333333,
+   "acc_at_50": 1.0,
+   "chance": 0.5,
+   "mean_conf": 0.9769946336746216
+  },
+  "tweet_offensive": {
+   "n": 860,
+   "heldout": false,
+   "acc": 0.8511627906976744,
+   "nll": 0.3344406485557556,
+   "brier": 0.20971718430519104,
+   "ece": 0.039765768203624464,
+   "aurc": 0.041883052711596785,
+   "acc_at_80": 0.9171511627906976,
+   "acc_at_50": 0.9767441860465116,
+   "chance": 0.5,
+   "mean_conf": 0.8272612690925598
+  },
+  "tweet_hate": {
+   "n": 1500,
+   "heldout": false,
+   "acc": 0.564,
+   "nll": 1.0489171743392944,
+   "brier": 0.6417430639266968,
+   "ece": 0.2539481555223465,
+   "aurc": 0.3877638717389759,
+   "acc_at_80": 0.5816666666666667,
+   "acc_at_50": 0.5893333333333334,
+   "chance": 0.5,
+   "mean_conf": 0.8179481625556946
+  },
+  "hate_offensive": {
+   "n": 1500,
+   "heldout": false,
+   "acc": 0.9266666666666666,
+   "nll": 0.2252429574728012,
+   "brier": 0.11920014768838882,
+   "ece": 0.018018441379070264,
+   "aurc": 0.016734557157267283,
+   "acc_at_80": 0.9741666666666666,
+   "acc_at_50": 0.9906666666666667,
+   "chance": 0.3333333333333333,
+   "mean_conf": 0.9164168238639832
+  },
+  "civil_comments": {
+   "n": 7500,
+   "heldout": false,
+   "acc": 0.9409333333333333,
+   "nll": 0.14164941012859344,
+   "brier": 0.08587079495191574,
+   "ece": 0.0033977080663044872,
+   "aurc": 0.0068072070084911175,
+   "acc_at_80": 0.9895,
+   "acc_at_50": 0.9994666666666666,
+   "chance": 0.5,
+   "mean_conf": 0.9395306706428528
+  },
+  "toxic_chat": {
+   "n": 3000,
+   "heldout": false,
+   "acc": 0.9843333333333333,
+   "nll": 0.04290607571601868,
+   "brier": 0.023861009627580643,
+   "ece": 0.005654305239518452,
+   "aurc": 0.0006420751251156756,
+   "acc_at_80": 0.9995833333333334,
+   "acc_at_50": 1.0,
+   "chance": 0.5,
+   "mean_conf": 0.9809242486953735
+  },
+  "sms_spam": {
+   "n": 1000,
+   "heldout": false,
+   "acc": 0.994,
+   "nll": 0.02838769368827343,
+   "brier": 0.01269171480089426,
+   "ece": 0.0056210076212883115,
+   "aurc": 0.00042434280645942466,
+   "acc_at_80": 0.99875,
+   "acc_at_50": 1.0,
+   "chance": 0.5,
+   "mean_conf": 0.9934377670288086
+  },
+  "enron_spam": {
+   "n": 1500,
+   "heldout": false,
+   "acc": 0.9953333333333333,
+   "nll": 0.014212104491889477,
+   "brier": 0.008566728793084621,
+   "ece": 0.004383029421170538,
+   "aurc": 6.272064980121389e-05,
+   "acc_at_80": 1.0,
+   "acc_at_50": 1.0,
+   "chance": 0.5,
+   "mean_conf": 0.994971513748169
+  },
+  "insincere_questions": {
+   "n": 1500,
+   "heldout": false,
+   "acc": 0.958,
+   "nll": 0.09950008988380432,
+   "brier": 0.058815646916627884,
+   "ece": 0.009067946950594569,
+   "aurc": 0.0033202849523762314,
+   "acc_at_80": 0.9966666666666667,
+   "acc_at_50": 1.0,
+   "chance": 0.5,
+   "mean_conf": 0.9590474963188171
+  },
+  "ade": {
+   "n": 1500,
+   "heldout": true,
+   "acc": 0.8373333333333334,
+   "nll": 0.3693406283855438,
+   "brier": 0.23190657794475555,
+   "ece": 0.035113215565681456,
+   "aurc": 0.054497040404324716,
+   "acc_at_80": 0.8941666666666667,
+   "acc_at_50": 0.96,
+   "chance": 0.5,
+   "mean_conf": 0.807280421257019
+  },
+  "snli": {
+   "n": 1476,
+   "heldout": false,
+   "acc": 0.9376693766937669,
+   "nll": 0.18385177850723267,
+   "brier": 0.0963604748249054,
+   "ece": 0.02269113342451853,
+   "aurc": 0.011467612685230727,
+   "acc_at_80": 0.9856054191363252,
+   "acc_at_50": 0.9932249322493225,
+   "chance": 0.3333333333333333,
+   "mean_conf": 0.9178707599639893
+  },
+  "mnli": {
+   "n": 1500,
+   "heldout": false,
+   "acc": 0.92,
+   "nll": 0.22414781153202057,
+   "brier": 0.12338652461767197,
+   "ece": 0.017153067986170414,
+   "aurc": 0.01603407033360699,
+   "acc_at_80": 0.9716666666666667,
+   "acc_at_50": 0.992,
+   "chance": 0.3333333333333333,
+   "mean_conf": 0.9121871590614319
+  },
+  "rte": {
+   "n": 277,
+   "heldout": false,
+   "acc": 0.9350180505415162,
+   "nll": 0.18287278711795807,
+   "brier": 0.10617374628782272,
+   "ece": 0.017879047118369462,
+   "aurc": 0.014183728145915892,
+   "acc_at_80": 0.9774774774774775,
+   "acc_at_50": 0.9927536231884058,
+   "chance": 0.5,
+   "mean_conf": 0.9323904514312744
+  },
+  "qnli": {
+   "n": 1500,
+   "heldout": false,
+   "acc": 0.9693333333333334,
+   "nll": 0.10291172564029694,
+   "brier": 0.05302723869681358,
+   "ece": 0.025760468522707608,
+   "aurc": 0.0035364855733933526,
+   "acc_at_80": 0.9975,
+   "acc_at_50": 0.9986666666666667,
+   "chance": 0.5,
+   "mean_conf": 0.9482638239860535
+  },
+  "qqp": {
+   "n": 1500,
+   "heldout": false,
+   "acc": 0.8553333333333333,
+   "nll": 0.29866960644721985,
+   "brier": 0.19174186885356903,
+   "ece": 0.02773328121503195,
+   "aurc": 0.033885104827278134,
+   "acc_at_80": 0.9275,
+   "acc_at_50": 0.9826666666666667,
+   "chance": 0.5,
+   "mean_conf": 0.8796607851982117
+  },
+  "mrpc": {
+   "n": 408,
+   "heldout": false,
+   "acc": 0.8872549019607843,
+   "nll": 0.26749035716056824,
+   "brier": 0.16417920589447021,
+   "ece": 0.022229443724248905,
+   "aurc": 0.027313885803315112,
+   "acc_at_80": 0.941717791411043,
+   "acc_at_50": 0.9852941176470589,
+   "chance": 0.5,
+   "mean_conf": 0.8898838758468628
+  },
+  "paws": {
+   "n": 1500,
+   "heldout": true,
+   "acc": 0.7293333333333333,
+   "nll": 0.7380470633506775,
+   "brier": 0.4195193648338318,
+   "ece": 0.16848981821537015,
+   "aurc": 0.15006823722693508,
+   "acc_at_80": 0.7791666666666667,
+   "acc_at_50": 0.8613333333333333,
+   "chance": 0.5,
+   "mean_conf": 0.8978231549263
+  },
+  "cola": {
+   "n": 1043,
+   "heldout": false,
+   "acc": 0.8609779482262704,
+   "nll": 0.30430471897125244,
+   "brier": 0.1895570456981659,
+   "ece": 0.03202069451459189,
+   "aurc": 0.035469170008069956,
+   "acc_at_80": 0.9268585131894485,
+   "acc_at_50": 0.975095785440613,
+   "chance": 0.5,
+   "mean_conf": 0.8630536794662476
+  },
+  "boolq": {
+   "n": 1500,
+   "heldout": false,
+   "acc": 0.9206666666666666,
+   "nll": 0.21292415261268616,
+   "brier": 0.12108734250068665,
+   "ece": 0.015688150127728763,
+   "aurc": 0.01887893970259577,
+   "acc_at_80": 0.965,
+   "acc_at_50": 0.988,
+   "chance": 0.5,
+   "mean_conf": 0.9252172112464905
+  },
+  "strategyqa": {
+   "n": 687,
+   "heldout": true,
+   "acc": 0.7394468704512372,
+   "nll": 0.518520712852478,
+   "brier": 0.34887009859085083,
+   "ece": 0.03629209233510271,
+   "aurc": 0.13807672950510763,
+   "acc_at_80": 0.78,
+   "acc_at_50": 0.8604651162790697,
+   "chance": 0.5,
+   "mean_conf": 0.7302948236465454
+  },
+  "pubmedqa": {
+   "n": 500,
+   "heldout": true,
+   "acc": 0.82,
+   "nll": 0.5464581251144409,
+   "brier": 0.29492902755737305,
+   "ece": 0.07814770770072936,
+   "aurc": 0.07736585822989323,
+   "acc_at_80": 0.8725,
+   "acc_at_50": 0.916,
+   "chance": 0.33333333333333326,
+   "mean_conf": 0.7438270449638367
+  },
+  "arc": {
+   "n": 1172,
+   "heldout": false,
+   "acc": 0.9641638225255973,
+   "nll": 0.11465426534414291,
+   "brier": 0.052921112626791,
+   "ece": 0.028441500261985424,
+   "aurc": 0.0038676297457262325,
+   "acc_at_80": 0.9936034115138592,
+   "acc_at_50": 0.9982935153583617,
+   "chance": 0.25015642775881686,
+   "mean_conf": 0.9453142881393433
+  },
+  "commonsense_qa": {
+   "n": 1221,
+   "heldout": false,
+   "acc": 0.8705978705978706,
+   "nll": 0.33728107810020447,
+   "brier": 0.18371696770191193,
+   "ece": 0.022902015874270598,
+   "aurc": 0.027533759986903508,
+   "acc_at_80": 0.9437052200614124,
+   "acc_at_50": 0.9836065573770492,
+   "chance": 0.19999999999999998,
+   "mean_conf": 0.8542661666870117
+  },
+  "qasc": {
+   "n": 926,
+   "heldout": false,
+   "acc": 0.9146868250539957,
+   "nll": 0.28406572341918945,
+   "brier": 0.13141772150993347,
+   "ece": 0.05652497216383276,
+   "aurc": 0.011926056294752855,
+   "acc_at_80": 0.9811066126855601,
+   "acc_at_50": 0.9978401727861771,
+   "chance": 0.125,
+   "mean_conf": 0.8592483401298523
+  },
+  "openbookqa": {
+   "n": 500,
+   "heldout": false,
+   "acc": 0.962,
+   "nll": 0.15028874576091766,
+   "brier": 0.062103498727083206,
+   "ece": 0.029689819455146767,
+   "aurc": 0.008756976491078847,
+   "acc_at_80": 0.9825,
+   "acc_at_50": 0.996,
+   "chance": 0.25,
+   "mean_conf": 0.9473608136177063
+  },
+  "sciq": {
+   "n": 1000,
+   "heldout": true,
+   "acc": 0.993,
+   "nll": 0.026743656024336815,
+   "brier": 0.011477932333946228,
+   "ece": 0.010730246603488931,
+   "aurc": 0.00012297224385520662,
+   "acc_at_80": 1.0,
+   "acc_at_50": 1.0,
+   "chance": 0.25,
+   "mean_conf": 0.9845147132873535
+  },
+  "hellaswag": {
+   "n": 1500,
+   "heldout": false,
+   "acc": 0.9693333333333334,
+   "nll": 0.07776311039924622,
+   "brier": 0.042344722896814346,
+   "ece": 0.006844995915889716,
+   "aurc": 0.0013723910741853661,
+   "acc_at_80": 1.0,
+   "acc_at_50": 1.0,
+   "chance": 0.25,
+   "mean_conf": 0.9741500616073608
+  },
+  "piqa": {
+   "n": 1500,
+   "heldout": false,
+   "acc": 0.954,
+   "nll": 0.12870272994041443,
+   "brier": 0.07193071395158768,
+   "ece": 0.014045794884363829,
+   "aurc": 0.006066841189670484,
+   "acc_at_80": 0.9891666666666666,
+   "acc_at_50": 0.9986666666666667,
+   "chance": 0.5,
+   "mean_conf": 0.9444655179977417
+  },
+  "social_iqa": {
+   "n": 1500,
+   "heldout": true,
+   "acc": 0.8233333333333334,
+   "nll": 0.4479636251926422,
+   "brier": 0.2568584680557251,
+   "ece": 0.02492147314548492,
+   "aurc": 0.06027574211898331,
+   "acc_at_80": 0.8875,
+   "acc_at_50": 0.9493333333333334,
+   "chance": 0.3333333333333333,
+   "mean_conf": 0.817057192325592
+  },
+  "winogrande": {
+   "n": 1267,
+   "heldout": false,
+   "acc": 0.8918705603788477,
+   "nll": 0.2955634295940399,
+   "brier": 0.16731061041355133,
+   "ece": 0.0350758349603805,
+   "aurc": 0.03663538162242224,
+   "acc_at_80": 0.9447731755424064,
+   "acc_at_50": 0.973186119873817,
+   "chance": 0.5,
+   "mean_conf": 0.9254278540611267
+  },
+  "race": {
+   "n": 1500,
+   "heldout": false,
+   "acc": 0.932,
+   "nll": 0.229837566614151,
+   "brier": 0.10741978138685226,
+   "ece": 0.012891520102818818,
+   "aurc": 0.015083615590388144,
+   "acc_at_80": 0.9783333333333334,
+   "acc_at_50": 0.9906666666666667,
+   "chance": 0.25,
+   "mean_conf": 0.935739278793335
+  },
+  "mmlu": {
+   "n": 1500,
+   "heldout": false,
+   "acc": 0.82,
+   "nll": 0.49843254685401917,
+   "brier": 0.24768584966659546,
+   "ece": 0.02976374606291451,
+   "aurc": 0.05049327310892201,
+   "acc_at_80": 0.91,
+   "acc_at_50": 0.9693333333333334,
+   "chance": 0.25,
+   "mean_conf": 0.8007211685180664
+  },
+  "medqa": {
+   "n": 1273,
+   "heldout": false,
+   "acc": 0.8546739984289081,
+   "nll": 0.39887696504592896,
+   "brier": 0.20918008685112,
+   "ece": 0.019769610706877737,
+   "aurc": 0.034844860161640345,
+   "acc_at_80": 0.9322200392927309,
+   "acc_at_50": 0.9874213836477987,
+   "chance": 0.25,
+   "mean_conf": 0.8520532846450806
+  },
+  "truthfulqa": {
+   "n": 817,
+   "heldout": true,
+   "acc": 0.7539779681762546,
+   "nll": 0.8089373111724854,
+   "brier": 0.364557683467865,
+   "ece": 0.06832761250919638,
+   "aurc": 0.08947353158858118,
+   "acc_at_80": 0.8425076452599388,
+   "acc_at_50": 0.9387254901960784,
+   "chance": 0.22622350449767833,
+   "mean_conf": 0.696648895740509
+  },
+  "fin_phrasebank": {
+   "n": 970,
+   "heldout": true,
+   "acc": 0.7587628865979381,
+   "nll": 0.6080690026283264,
+   "brier": 0.3605556786060333,
+   "ece": 0.10978183436025048,
+   "aurc": 0.11749813303878813,
+   "acc_at_80": 0.8195876288659794,
+   "acc_at_50": 0.8886597938144329,
+   "chance": 0.33333333333333326,
+   "mean_conf": 0.6539795398712158
+  },
+  "bias_in_bios": {
+   "n": 3000,
+   "heldout": false,
+   "acc": 0.9666666666666667,
+   "nll": 0.09920930117368698,
+   "brier": 0.04930334910750389,
+   "ece": 0.008911347289880108,
+   "aurc": 0.00193519783058256,
+   "acc_at_80": 0.9991666666666666,
+   "acc_at_50": 1.0,
+   "chance": 0.30000000000000004,
+   "mean_conf": 0.9638564586639404
+  },
+  "helpsteer2": {
+   "n": 5190,
+   "heldout": false,
+   "acc": 0.6163776493256262,
+   "nll": 0.8915571570396423,
+   "brier": 0.49822765588760376,
+   "ece": 0.01934157278037023,
+   "aurc": 0.2413207825095569,
+   "acc_at_80": 0.6763005780346821,
+   "acc_at_50": 0.7533718689788054,
+   "chance": 0.2,
+   "mean_conf": 0.620047390460968
+  },
+  "helpsteer3_pref": {
+   "n": 1176,
+   "heldout": false,
+   "acc": 0.5518707482993197,
+   "nll": 1.2033027410507202,
+   "brier": 0.5932223796844482,
+   "ece": 0.025830757785208366,
+   "aurc": 0.30366389562151924,
+   "acc_at_80": 0.6068012752391073,
+   "acc_at_50": 0.6768707482993197,
+   "chance": 0.14285714285714282,
+   "mean_conf": 0.5695160031318665
+  },
+  "hate_speech_scales": {
+   "n": 7500,
+   "heldout": false,
+   "acc": 0.5805333333333333,
+   "nll": 0.9507390856742859,
+   "brier": 0.528562068939209,
+   "ece": 0.013830039529005684,
+   "aurc": 0.2502365025510708,
+   "acc_at_80": 0.6266666666666667,
+   "acc_at_50": 0.7202666666666667,
+   "chance": 0.22666666666666674,
+   "mean_conf": 0.5702805519104004
+  },
+  "liar2": {
+   "n": 1500,
+   "heldout": false,
+   "acc": 0.422,
+   "nll": 1.3505219221115112,
+   "brier": 0.6785846948623657,
+   "ece": 0.028518199036518737,
+   "aurc": 0.4162147525556497,
+   "acc_at_80": 0.4708333333333333,
+   "acc_at_50": 0.5653333333333334,
+   "chance": 0.16666666666666666,
+   "mean_conf": 0.41971758008003235
+  },
+  "prosocial_safety": {
+   "n": 1500,
+   "heldout": false,
+   "acc": 0.528,
+   "nll": 1.1013150215148926,
+   "brier": 0.5662647485733032,
+   "ece": 0.030879737019538868,
+   "aurc": 0.2518870991484317,
+   "acc_at_80": 0.5991666666666666,
+   "acc_at_50": 0.7213333333333334,
+   "chance": 0.2,
+   "mean_conf": 0.518919050693512
+  },
+  "ultrafeedback_pref": {
+   "n": 1500,
+   "heldout": false,
+   "acc": 0.784,
+   "nll": 0.43619513511657715,
+   "brier": 0.28585085272789,
+   "ece": 0.03089568722248076,
+   "aurc": 0.08515379420314682,
+   "acc_at_80": 0.8475,
+   "acc_at_50": 0.9266666666666666,
+   "chance": 0.5,
+   "mean_conf": 0.8054375648498535
+  },
+  "shp": {
+   "n": 1500,
+   "heldout": false,
+   "acc": 0.79,
+   "nll": 0.44603705406188965,
+   "brier": 0.2889350950717926,
+   "ece": 0.02012235975265503,
+   "aurc": 0.09319282571857365,
+   "acc_at_80": 0.8491666666666666,
+   "acc_at_50": 0.9146666666666666,
+   "chance": 0.5,
+   "mean_conf": 0.7964972257614136
+  },
+  "hh_rlhf": {
+   "n": 1488,
+   "heldout": false,
+   "acc": 0.6901881720430108,
+   "nll": 0.5672697424888611,
+   "brier": 0.38868722319602966,
+   "ece": 0.018854505592776898,
+   "aurc": 0.18054136093942436,
+   "acc_at_80": 0.7361344537815127,
+   "acc_at_50": 0.8077956989247311,
+   "chance": 0.5,
+   "mean_conf": 0.7015084028244019
+  },
+  "arena_pref": {
+   "n": 1500,
+   "heldout": true,
+   "acc": 0.5206666666666667,
+   "nll": 1.0640077590942383,
+   "brier": 0.6215369701385498,
+   "ece": 0.1206609118382136,
+   "aurc": 0.36246136582814387,
+   "acc_at_80": 0.55,
+   "acc_at_50": 0.616,
+   "chance": 0.3333333333333333,
+   "mean_conf": 0.6390127539634705
+  },
+  "reward_bench": {
+   "n": 1500,
+   "heldout": true,
+   "acc": 0.9186666666666666,
+   "nll": 0.2031487673521042,
+   "brier": 0.12078241258859634,
+   "ece": 0.023855606953302997,
+   "aurc": 0.014145065893525021,
+   "acc_at_80": 0.9758333333333333,
+   "acc_at_50": 0.9946666666666667,
+   "chance": 0.5,
+   "mean_conf": 0.8953665494918823
+  },
+  "glaive_tools": {
+   "n": 1348,
+   "heldout": false,
+   "acc": 0.9532640949554896,
+   "nll": 0.14300261437892914,
+   "brier": 0.07691621780395508,
+   "ece": 0.02094891802228167,
+   "aurc": 0.007086222936904486,
+   "acc_at_80": 0.9860853432282004,
+   "acc_at_50": 0.9985163204747775,
+   "chance": 0.16666666666666666,
+   "mean_conf": 0.9427018165588379
+  },
+  "toolace": {
+   "n": 1000,
+   "heldout": false,
+   "acc": 0.94,
+   "nll": 0.1773465871810913,
+   "brier": 0.08758451044559479,
+   "ece": 0.019288902848959,
+   "aurc": 0.01419978829962143,
+   "acc_at_80": 0.98125,
+   "acc_at_50": 0.992,
+   "chance": 0.12470555555555554,
+   "mean_conf": 0.9342852830886841
+  },
+  "hermes_tools": {
+   "n": 1500,
+   "heldout": true,
+   "acc": 0.7986666666666666,
+   "nll": 0.36825039982795715,
+   "brier": 0.23031076788902283,
+   "ece": 0.08474178238709767,
+   "aurc": 0.04541348669345317,
+   "acc_at_80": 0.8966666666666666,
+   "acc_at_50": 0.988,
+   "chance": 0.125,
+   "mean_conf": 0.8443251848220825
+  },
+  "copa": {
+   "n": 100,
+   "heldout": false,
+   "acc": 1.0,
+   "nll": 0.018447600305080414,
+   "brier": 0.004432467743754387,
+   "ece": 0.017155210971832275,
+   "aurc": 0.0,
+   "acc_at_80": 1.0,
+   "acc_at_50": 1.0,
+   "chance": 0.5,
+   "mean_conf": 0.9828447699546814
+  },
+  "wic": {
+   "n": 638,
+   "heldout": false,
+   "acc": 0.7727272727272727,
+   "nll": 0.47851547598838806,
+   "brier": 0.3126711845397949,
+   "ece": 0.06492637840557994,
+   "aurc": 0.10600177937132382,
+   "acc_at_80": 0.8352941176470589,
+   "acc_at_50": 0.8996865203761756,
+   "chance": 0.5,
+   "mean_conf": 0.828840970993042
+  },
+  "multirc": {
+   "n": 1500,
+   "heldout": false,
+   "acc": 0.918,
+   "nll": 0.23172627389431,
+   "brier": 0.1296291947364807,
+   "ece": 0.02276274232069647,
+   "aurc": 0.02589494647653243,
+   "acc_at_80": 0.965,
+   "acc_at_50": 0.9773333333333334,
+   "chance": 0.5,
+   "mean_conf": 0.9249215722084045
+  },
+  "cb": {
+   "n": 56,
+   "heldout": true,
+   "acc": 0.8928571428571429,
+   "nll": 0.2884705662727356,
+   "brier": 0.15287914872169495,
+   "ece": 0.08375836323414529,
+   "aurc": 0.01340165852858295,
+   "acc_at_80": 0.9777777777777777,
+   "acc_at_50": 1.0,
+   "chance": 0.3333333333333333,
+   "mean_conf": 0.8508859276771545
+  },
+  "fever": {
+   "n": 1500,
+   "heldout": false,
+   "acc": 0.91,
+   "nll": 0.2639821767807007,
+   "brier": 0.1403927505016327,
+   "ece": 0.01887864359219866,
+   "aurc": 0.021236299620246823,
+   "acc_at_80": 0.9666666666666667,
+   "acc_at_50": 0.9866666666666667,
+   "chance": 0.3333333333333333,
+   "mean_conf": 0.911332368850708
+  },
+  "wiki_qa": {
+   "n": 879,
+   "heldout": false,
+   "acc": 0.9089874857792947,
+   "nll": 0.22839467227458954,
+   "brier": 0.13368310034275055,
+   "ece": 0.023650864081984878,
+   "aurc": 0.020796820711007552,
+   "acc_at_80": 0.9644381223328592,
+   "acc_at_50": 0.9886363636363636,
+   "chance": 0.5,
+   "mean_conf": 0.8948045372962952
+  },
+  "msmarco_rel": {
+   "n": 1446,
+   "heldout": false,
+   "acc": 0.6970954356846473,
+   "nll": 0.5717816352844238,
+   "brier": 0.39096301794052124,
+   "ece": 0.03819479437784535,
+   "aurc": 0.18340211744146934,
+   "acc_at_80": 0.7303370786516854,
+   "acc_at_50": 0.8091286307053942,
+   "chance": 0.5,
+   "mean_conf": 0.7234166860580444
+  },
+  "medmcqa": {
+   "n": 1500,
+   "heldout": false,
+   "acc": 0.742,
+   "nll": 0.6534839272499084,
+   "brier": 0.3494548797607422,
+   "ece": 0.04476275533437729,
+   "aurc": 0.09236420121984344,
+   "acc_at_80": 0.8175,
+   "acc_at_50": 0.932,
+   "chance": 0.25,
+   "mean_conf": 0.7632542848587036
+  },
+  "quality": {
+   "n": 1500,
+   "heldout": true,
+   "acc": 0.632,
+   "nll": 0.9736621379852295,
+   "brier": 0.5085250735282898,
+   "ece": 0.09598391598463062,
+   "aurc": 0.20000924342798884,
+   "acc_at_80": 0.69,
+   "acc_at_50": 0.7893333333333333,
+   "chance": 0.25,
+   "mean_conf": 0.726018488407135
+  },
+  "xstory_cloze": {
+   "n": 1500,
+   "heldout": true,
+   "acc": 0.9946666666666667,
+   "nll": 0.026280736550688744,
+   "brier": 0.011265838518738747,
+   "ece": 0.015819201707839965,
+   "aurc": 8.187136167601649e-05,
+   "acc_at_80": 1.0,
+   "acc_at_50": 1.0,
+   "chance": 0.5,
+   "mean_conf": 0.9801904559135437
+  },
+  "abstain_probe": {
+   "n": 1620,
+   "heldout": true,
+   "acc": 0.6222222222222222,
+   "nll": 1.1009931564331055,
+   "brier": 0.5079091191291809,
+   "ece": 0.08518773355969675,
+   "aurc": 0.17190129595658887,
+   "acc_at_80": 0.7106481481481481,
+   "acc_at_50": 0.8222222222222222,
+   "chance": 0.17897217323143252,
+   "mean_conf": 0.7074099779129028
+  },
+  "agenttraj": {
+   "n": 1500,
+   "heldout": false,
+   "acc": 0.9366666666666666,
+   "nll": 0.15657173097133636,
+   "brier": 0.08615637570619583,
+   "ece": 0.018042015492916074,
+   "aurc": 0.004964706283705984,
+   "acc_at_80": 0.9958333333333333,
+   "acc_at_50": 0.9986666666666667,
+   "chance": 0.2,
+   "mean_conf": 0.9253047108650208
+  },
+  "mind2web": {
+   "n": 1500,
+   "heldout": false,
+   "acc": 0.8853333333333333,
+   "nll": 0.33396658301353455,
+   "brier": 0.1709548383951187,
+   "ece": 0.015519337852795898,
+   "aurc": 0.02253901936947198,
+   "acc_at_80": 0.9558333333333333,
+   "acc_at_50": 0.9933333333333333,
+   "chance": 0.16666666666666666,
+   "mean_conf": 0.885744035243988
+  },
+  "synth": {
+   "n": 300,
+   "heldout": false,
+   "acc": 0.8466666666666667,
+   "nll": 0.39028388261795044,
+   "brier": 0.21442179381847382,
+   "ece": 0.03632553676764172,
+   "aurc": 0.041584943277947255,
+   "acc_at_80": 0.9166666666666666,
+   "acc_at_50": 0.9733333333333334,
+   "chance": 0.37149999999999994,
+   "mean_conf": 0.8786032199859619
+  },
+  "offtopic_probe": {
+   "n": 1620,
+   "heldout": true,
+   "acc": 0.8697530864197531,
+   "nll": 0.3856513202190399,
+   "brier": 0.17581190168857574,
+   "ece": 0.037777107972421745,
+   "aurc": 0.018382871739502517,
+   "acc_at_80": 0.9683641975308642,
+   "acc_at_50": 0.9950617283950617,
+   "chance": 0.17979423868312755,
+   "mean_conf": 0.8545409440994263
+  },
+  "hwu64": {
+   "n": 1076,
+   "heldout": true,
+   "acc": 0.974907063197026,
+   "nll": 0.08065392076969147,
+   "brier": 0.04049574211239815,
+   "ece": 0.021740955629534923,
+   "aurc": 0.0012811690911756525,
+   "acc_at_80": 1.0,
+   "acc_at_50": 1.0,
+   "chance": 0.09999999999999999,
+   "mean_conf": 0.9563130140304565
+  },
+  "dbpedia_l2": {
+   "n": 1500,
+   "heldout": true,
+   "acc": 0.9606666666666667,
+   "nll": 0.11009299755096436,
+   "brier": 0.060155414044857025,
+   "ece": 0.01014547139406203,
+   "aurc": 0.003270819231754314,
+   "acc_at_80": 0.9958333333333333,
+   "acc_at_50": 1.0,
+   "chance": 0.1,
+   "mean_conf": 0.9609081149101257
+  },
+  "dbpedia_l3": {
+   "n": 1500,
+   "heldout": true,
+   "acc": 0.992,
+   "nll": 0.02752424031496048,
+   "brier": 0.013588163070380688,
+   "ece": 0.0035614574154217704,
+   "aurc": 0.000210942320122308,
+   "acc_at_80": 1.0,
+   "acc_at_50": 1.0,
+   "chance": 0.1,
+   "mean_conf": 0.9927933216094971
+  },
+  "quality_full": {
+   "n": 600,
+   "heldout": true,
+   "acc": 0.565,
+   "nll": 1.071710467338562,
+   "brier": 0.5663425922393799,
+   "ece": 0.11241838062802949,
+   "aurc": 0.2527098290509578,
+   "acc_at_80": 0.6270833333333333,
+   "acc_at_50": 0.7366666666666667,
+   "chance": 0.25,
+   "mean_conf": 0.6741220951080322
+  }
+ },
+ "other_sets": {
+  "general_validation_847": {
+   "acc": 0.8996,
+   "nll": 0.3285,
+   "ece": 0.0442
+  },
+  "typesafe_102": {
+   "acc": 0.8627,
+   "nll": 0.3415,
+   "ece": 0.065,
+   "tv_to_reference": 0.1648
+  },
+  "openjev_5252": {
+   "acc": 0.6828,
+   "macro_f1": 0.8567,
+   "nll": 0.7523,
+   "ece": 0.0491
+  },
+  "mind2web_1770": {
+   "acc": 0.8955,
+   "nll": 0.3159,
+   "ece": 0.0571
+  },
+  "jevbench_public_items": {
+   "easy": 1.0,
+   "standard": 0.9722,
+   "hard": 0.6757
+  },
+  "bespoke_public_suite": {
+   "macro": 0.7737,
+   "micro": 0.7866,
+   "macro_untrained": 0.7435
+  },
+  "miniwob_live_22x8": {
+   "sampled": 0.8636,
+   "sampled_held_out": 0.7917,
+   "greedy": 0.9716,
+   "greedy_held_out": 0.9792
+  },
+  "zero_shot_games_win_rate": {
+   "sampled_all": 0.2415,
+   "greedy_all": 0.3718
+  }
+ }
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "eos_token_id": 248044,
+  "transformers_version": "5.17.0",
+  "use_cache": true
+}

model-00001-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c230ef13988becedf0c3f81c76628283663007466d8bfcead27bd2d85a297935
+size 4861004088

model-00002-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4eaeb99c73f9d075d6aff9f2245fe88b7e3b5faf546b4c110316174fa61e77f5
+size 4506714376

model-00003-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f31fa5d8cd003aed7b8fb2b428d90ba9d3d7e12071fc0c07b17b0f67bbdee140
+size 4989093232

model-00004-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4407b99000e95cc906268010703e0036976fe015425fbd160b28c56f7008bfed
+size 4574152520

model-00005-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5e5eed694729abdf8dc0eb22580ee724501040e71ab65c5b688f125c1ecf0006
+size 4976513672

model-00006-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ee091bd7107b1e5c347153bc116ba385816c2b67f045c6ab59f38ae18d266280
+size 4990860936

model-00007-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0d540f11b3df3134db58830f8d8c6979a5504c89e48c0ebc90928f321184e26d
+size 4564708264

model-00008-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5a7062793d00efb2d8904198968d52159cc2c2bbad03886020395db0aae0c136
+size 4506716424

model-00009-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d806e89b85bf1a360b9c576f6de907e67f41b4443034d261889b65ba478efaf4
+size 4506716432

model-00010-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ceebc6759cae88ad2fa75c0d4f67a3c83bfb23c540ec546a6d661193388d7cba
+size 4989095576

model-00011-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5a329b4b880be2bd3526214c8c068d9dad3b32d5ba497e054444e761951f0334
+size 4499363680

model-00012-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4a49900cb63a3bd7a1e8a9d4fb127d5283e5243a8a32aef92326e456ad9d0677
+size 4506716472

model-00013-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8dde20ac69c397bbdfafc78355032278878d60c860ecd1622e74c37676cbe628
+size 4989095576

model-00014-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3ec1e988ee302b3e5bd282f93e18a741c69fec77f2021b397f2478145cdc89b9
+size 4574154552

model-00015-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1aa8bef406b36ed1479c2e50adb568a1367de94d0bc622a3b8809b4625494cbb
+size 3290673624

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523
+size 19989325

tokenizer_config.json

@@ -0,0 +1,32 @@
+{
+  "add_prefix_space": false,
+  "audio_bos_token": "<|audio_start|>",
+  "audio_eos_token": "<|audio_end|>",
+  "audio_token": "<|audio_pad|>",
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "image_token": "<|image_pad|>",
+  "is_local": true,
+  "local_files_only": true,
+  "model_max_length": 262144,
+  "model_specific_special_tokens": {
+    "audio_bos_token": "<|audio_start|>",
+    "audio_eos_token": "<|audio_end|>",
+    "audio_token": "<|audio_pad|>",
+    "image_token": "<|image_pad|>",
+    "video_token": "<|video_pad|>",
+    "vision_bos_token": "<|vision_start|>",
+    "vision_eos_token": "<|vision_end|>"
+  },
+  "pad_token": "<|endoftext|>",
+  "pretokenize_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null,
+  "video_token": "<|video_pad|>",
+  "vision_bos_token": "<|vision_start|>",
+  "vision_eos_token": "<|vision_end|>"
+}