cloudyu commited on
Commit
4ab5dfb
·
verified ·
1 Parent(s): 3b960eb

Model card: aligned with JEV-27B — first integrated System 1 + System 2 model, Blocks of Experts recipe, KL fidelity, speed vs hosted API, comparison with other reproductions

Browse files
Files changed (1) hide show
  1. README.md +384 -220
README.md CHANGED
@@ -10,6 +10,8 @@ library_name: transformers
10
  pipeline_tag: text-classification
11
  tags:
12
  - system-one
 
 
13
  - typed-decisions
14
  - decision-model
15
  - calibrated-probabilities
@@ -40,13 +42,13 @@ model-index:
40
  split: test_set_30k
41
  metrics:
42
  - type: kl_divergence
43
- name: mean KL(teacher ‖ model)
44
  value: 0.0210
45
  - type: auroc
46
  name: noul AUROC
47
- value: 0.9958
48
  - type: brier
49
- name: noul Brier (vs. teacher probability)
50
  value: 0.0015
51
  - type: mae
52
  name: score expected-value MAE (0–5 scale)
@@ -58,68 +60,265 @@ model-index:
58
  name: choice top-1 agreement (all rows)
59
  value: 0.898
60
  - type: accuracy
61
- name: choice top-1 agreement (teacher-decisive rows, top-2 gap ≥ 0.1)
62
  value: 0.954
 
 
 
 
 
 
 
 
 
 
 
63
  ---
64
 
65
- # autotrust/JEV-9B — an open-weights student of TypeSafe Jev 1.13 (Qwen3.5-9B, calibrated *System One* decisions)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
66
 
67
- > **Two different models, two different organisations — please read the names carefully.**
68
- > **TypeSafe Jev 1.13** is the hosted, closed model made by TypeSafe AI; it is the *teacher* whose public
69
- > outputs this model was trained on. **autotrust/JEV-9B** (this repository) is an independent open-weights
70
- > *student* built by autotrust from the Apache-2.0 corpus
71
- > [`SargeDev/jev-distill-corpus-v3`](https://huggingface.co/datasets/SargeDev/jev-distill-corpus-v3).
72
- > autotrust/JEV-9B is not affiliated with, endorsed by, or a product of TypeSafe AI. Throughout this card the
73
- > teacher is always written *TypeSafe Jev 1.13* and this model is always written *autotrust/JEV-9B*.
74
 
75
- **One forward pass in, a calibrated probability distribution out.** autotrust/JEV-9B answers typed questions
76
- (`noul` yes/no · `choice` over 2–16 options · `score` on a 0–5 scale) about any text or JSON state.
77
- No JSON parsing, no prompt engineering — it was distilled from the full output distributions of
78
- **TypeSafe Jev 1.13** onto a pretrained **Qwen3.5-9B** backbone. The bundle ships **two heads on one
79
- backbone**: the 24-slot decision head (typed decisions) and the original `lm_head` (ordinary text
80
- generation). The backbone weights on disk are **bit-identical to Qwen3.5-9B**; the decision adapter is a
81
- separate 154 MB LoRA that is only applied for decisions — so generation quality is exactly the base model's.
82
 
83
- **Model family** — same recipe, same API, same two-head packaging:
84
- **autotrust/JEV-9B** (this repository, Qwen3.5-9B, 18 GB) · [autotrust/JEV-27B](https://huggingface.co/autotrust/JEV-27B)
85
- (Qwen3.8-27B, 54 GB: KL 0.019, choice 90.3 %, out-of-distribution KL 0.104, HumanEval 78.0 %).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
86
 
87
- ## Headline results
88
 
89
- Held-out test set: 29,955 questions, 53 domains, all three question types. Numbers are for **autotrust/JEV-9B**.
 
 
 
 
90
 
91
- | What is measured | autotrust/JEV-9B | How to read it |
92
  |---|---|---|
93
- | **How closely it reproduces the teacher's probabilities** — mean KL divergence, lower is better, 0 = identical | **0.021** | The student's probability distributions are nearly indistinguishable from TypeSafe Jev 1.13's |
94
- | **Yes/no questions** (`noul`) — AUROC, higher is better | **0.996** | Ranks true vs. false almost perfectly |
95
- | **Multiple-choice questions** (`choice`) — picks the same option as the teacher | **89.8 %** | Over all questions, including those where the teacher itself is nearly tied |
96
- | **Multiple-choice questions where the teacher has a clear favourite** (its top two options differ by ≥ 0.1) | **95.4 %** | On the remaining, near-tie questions any faithful copy would agree only about half the time |
97
- | **Rating questions** (`score`, 0–5 scale) — mean error of the expected rating, lower is better | **0.10** | About one tenth of a rating step |
98
- | **Calibration** — expected calibration error, lower is better | **0.0007** | A stated 80 % is an 80 %; no post-hoc temperature correction was needed (fitted temperatures ≈ 1.00) |
99
- | **Tasks never seen in training** (Open-Jev OOD split) — mean KL divergence | **0.234** | Still tracks the teacher closely on unfamiliar task families |
100
- | **Independent benchmark with human labels** (intent / topic / emotion classification, not in our training data) — accuracy relative to TypeSafe Jev 1.13 | **97 %** with 2 options, **90 %** with 16 | Degrades faster than the teacher as the option list grows; [autotrust/JEV-27B](https://huggingface.co/autotrust/JEV-27B) keeps 96 % at 16. See [Tested on fresh, real-world data](#tested-on-fresh-real-world-data) |
101
- | **Text generation** (AR head) — HumanEval pass@1, greedy | **70.7 %** | Identical to Qwen3.5-9B (116/164, byte-identical completions): the generation path is untouched |
102
- | **Latency** — one request on one B200 GPU, median | **≈ 90 ms** | 2.5 ms per decision when requests are batched |
103
 
104
- Only **40.2 M parameters** (0.5 % of the 8 B backbone) were trained, for **≈ 3 GPU-hours** on a single
105
- B200 (0.93 epoch ≈ 608 k rows). Full evaluation, including robustness and per-domain breakdowns, is in
106
- [Evaluation details](#evaluation-details) and in `reports/eval_s2_9b_epoch1.md`.
107
 
108
- **Runs on vLLM** — one engine serves both heads from the same weights through an OpenAI-compatible API; text generation is ≈ 50× faster than the PyTorch path. See [Inference with vLLM](#inference-with-vllm-recommended).
109
 
110
- ## Tested on fresh, real-world data
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
111
 
112
- Two checks on inputs the model has never seen: an independent benchmark with human gold labels on which TypeSafe
113
- Jev 1.13's own scores are public, and a set of decisions on Hacker News and V2EX content from 23–25 September 2026.
114
- The 27 B sibling is shown for comparison.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
115
 
116
  ### Independent benchmark: side by side with TypeSafe Jev 1.13
117
 
118
  [`gazelle93/decision-models-under-pressure`](https://github.com/gazelle93/decision-models-under-pressure) (published
119
  25 Sep 2026) asks decision models to pick the right label for real texts from CLINC-150, MTOP, GoEmotions, DBpedia and
120
- financial tweets, under three kinds of pressure: more options, near-miss options, and shuffled option order. The labels
121
- are human gold labels, none of this data is in our training set, and TypeSafe Jev 1.13's results are published with
122
- it. We re-ran the same items, up to our 16-option limit.
123
 
124
  | | TypeSafe Jev 1.13 (published) | **autotrust/JEV-9B** | autotrust/JEV-27B |
125
  |---|---|---|---|
@@ -129,16 +328,15 @@ it. We re-ran the same items, up to our 16-option limit.
129
  | Answers changed by shuffling the options alone (16 options, 5 orderings) | 7.0 % | **11.5 %** | 7.4 % |
130
  | Time for 14,400 decisions on one B200 | — | **42 s** | 110 s |
131
 
132
- With few options the 9 B student is within 2–3 points of the teacher on data it was never trained on (97 % of its
133
- accuracy at 2 options). As the list grows it falls behind faster: 90 % of the teacher's accuracy at 16 options,
134
- against 96 % for the 27 B sibling. It is also more sensitive to option order (11.5 % of answers change vs 7.0 % for
135
- the teacher) and loses a little more on near-miss options. It is 2.6× faster than the 27 B model. Our run follows the
136
- benchmark's published method (gold plus the first K−1 distractors of a pool, shuffled per item); the orderings are
137
- seeded differently, so compare the aggregates, not individual items.
138
 
139
  ### Fresh examples (Hacker News and V2EX, 23–25 September 2026)
140
 
141
- The expected answers were written by hand before the model was run. These are illustrations (≈ 110 decisions), not a
142
  benchmark.
143
 
144
  | task | autotrust/JEV-9B | autotrust/JEV-27B |
@@ -152,52 +350,51 @@ benchmark.
152
  |---|---|---|
153
  | HN: "Two-tier encryption in the UK" | topic (10 options) | security & privacy · 0.87 |
154
  | HN: "Using LLMs to trace alchemical knowledge and decode 17th century letters" | about AI? | P(true) = 0.88 |
155
- | HN comment: "Why do you talk like that? Are you one of the redditors that has been invading…" | insults or attacks someone? | P(true) = 0.67 |
156
  | HN comment: "Please stop this. We've asked you before to observe the guidelines…" | what is it mainly doing? | moderating the discussion · 0.78 |
157
  | V2EX: "一个不需要 gemini pro 的完全免费��注册 Muse 的方法 … 邀请码:…" | contains a referral / invite code? | P(true) = 1.00 |
158
  | V2EX: "今天中秋节,还要加班的有吗?来报道下" | promotes a product or paid offer? | P(true) = 0.01 |
159
  | Diff replacing a parameterised query with `"… WHERE id = " + request.args["id"]` | introduces a security vulnerability? | P(true) = 0.93 (0.17 for a variable rename) |
160
  | "I'm not happy with the fit. What are my options here?" | asking for a refund? | P(true) = 0.17 (TypeSafe's docs report 0.22 for Jev on this exact text) |
161
 
162
- Where it failed or wavered — and where the 27 B sibling did better:
163
 
164
- * **Code-rule check**: it missed a TypeScript file that declares `const port: number = Number(process.env.PORT)`
165
- against the rule "a port must be a branded, range-checked integer" (0.33); autotrust/JEV-27B flags it (0.93).
166
- * **Fraud screening**: a CEO wire-transfer (business-email-compromise) message is flagged, but only at 0.56 (27 B: 0.84).
167
  * **Comment intent**: "Because it's not a real argument. It's a deflection people use." read as attacking another
168
  commenter (0.66) rather than arguing a point.
169
- * **A poker spot with the nuts** (a solver checks 100 %): shoves with 0.70; TypeSafe Jev 1.13 shoved with 0.62 in a
170
- published test, and the 27 B sibling with 0.63, so this mistake comes from the teacher.
171
-
172
- **Which size to pick:** for routing, moderation, topic triage and short option lists the 9 B model gives nearly the
173
- same answers 2.6× faster. For long option lists (more than about 8), code-rule checks and fraud screening, use
174
- [autotrust/JEV-27B](https://huggingface.co/autotrust/JEV-27B).
175
 
176
  Per-example outputs and the benchmark aggregates are in `reports/realworld_9b.json` (the HN and V2EX inputs came from
177
  their public APIs on 25 September 2026).
178
 
179
- ## Inference with vLLM (recommended)
180
 
181
- **One vLLM engine serves both heads from the same pristine weights.** Ordinary requests go through the base
182
- `lm_head` (text generation, exactly Qwen3.5-9B); requests addressed to the LoRA module `jev-decision` go through the
183
- decision head. `adapter_vllm/` contains the backbone LoRA plus the 24-slot decision head re-expressed as an
184
- `lm_head` LoRA (only the 24 verbalizer rows change), so a typed decision is a single prefill step with
185
- `max_tokens=1`, constrained to the option tokens and read back as log-probabilities.
186
 
187
  ### 1 — Start the server (OpenAI-compatible)
188
 
189
  ```bash
190
- hf download autotrust/JEV-9B --local-dir JEV-9B # ~18 GB
191
  vllm serve JEV-9B --served-model-name autotrust/JEV-9B \
192
  --enable-lora --max-lora-rank 32 --lora-modules jev-decision=JEV-9B/adapter_vllm \
193
  --logprobs-mode processed_logprobs --max-model-len 4096
194
  ```
195
 
196
- `--logprobs-mode processed_logprobs` is required: it makes the returned log-probabilities respect
197
- `allowed_token_ids`. Add `--enable-prefix-caching --mamba-cache-mode align` if you ask many questions about the same
198
- state (see notes below).
199
 
200
- ### 2 — Text generation: the unmodified base model
201
 
202
  ```bash
203
  curl -s localhost:8000/v1/chat/completions -H 'content-type: application/json' -d '{
@@ -206,7 +403,11 @@ curl -s localhost:8000/v1/chat/completions -H 'content-type: application/json' -
206
  "max_tokens": 60, "chat_template_kwargs": {"enable_thinking": false}}'
207
  ```
208
 
209
- ### 3 — Typed decisions: the decision head (Python, only `requests` + two small JSON files)
 
 
 
 
210
 
211
  ```python
212
  import json, math, requests
@@ -238,10 +439,31 @@ print(decide("noul", "Customer says the parcel arrived damaged and wants their m
238
  "Is the customer asking for a refund?"))
239
  ```
240
 
241
- Values can differ in the third decimal between runs: vLLM computes in bf16 and results depend slightly on
242
- which requests are batched together. The adding of the head bias and the division by the temperature happen client-side; the log-softmax normaliser
243
  that vLLM applies cancels out, so the result is exactly the decision head's calibrated distribution.
244
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
245
  ### Offline / batch (Python API)
246
 
247
  ```python
@@ -251,39 +473,38 @@ from vllm.lora.request import LoRARequest
251
  llm = LLM("JEV-9B", enable_lora=True, max_lora_rank=32, logprobs_mode="processed_logprobs", max_model_len=4096)
252
  decision = LoRARequest("jev-decision", 1, "JEV-9B/adapter_vllm")
253
 
254
- gen = llm.generate(["..."], SamplingParams(temperature=0.0, max_tokens=256)) # base model, no LoRA
255
- dec = llm.generate([prompt], [SamplingParams(max_tokens=1, temperature=1.0, # decision head
256
  allowed_token_ids=ids, logprobs=len(ids))],
257
  lora_request=decision) # then + bias, / T as above
258
  ```
259
 
260
- Mixed batches work too: pass a per-request `lora_request` list (`None` for generation, `decision` for
261
- decisions) and both heads are served in the same `generate` call.
262
 
263
  ### Measured on one B200
264
 
265
  | workload | PyTorch path | **vLLM** |
266
  |---|---|---|
267
- | Text generation — 164 HumanEval completions (greedy, ≤ 384 new tokens) | 165 s | **3.3 s** (≈ 50×) |
268
- | Typed decisions, offline batch — 29,955 test questions | 75 s (398 q/s) | 80 s (374 q/s) |
269
- | Typed decisions over HTTP — 64 / 256 concurrent clients | — | 150 / 205 req/s |
270
- | Decision fidelity vs. the PyTorch path | — | test KL 0.0211 (identical); mean \|Δp\| 0.0008 over HTTP |
271
  | Many questions about one state, `--enable-prefix-caching` | — | +14–20 % throughput |
272
 
273
  Notes:
274
- * The big win is on the generation head. Decisions are a single prefill pass with no decoding, so offline batch
275
- throughput is about the same as the PyTorch path; for decisions vLLM mainly buys serving: continuous batching
276
- under concurrency, an OpenAI-compatible API, and one engine for both heads.
277
- * Prefix caching: this architecture mixes Gated DeltaNet and attention layers, and vLLM caches it in blocks of
278
- 528 tokens, so only shared prefixes longer than 528 tokens are reused. The template puts `[kind]` before
279
- `[state]`, so only questions of the same kind share a prefix. On 293 real states × 7.7 yes/no questions each
280
- (≈ 480-token states) prefix caching served 19.6 % of prompt tokens from cache (+14–20 % throughput) with
281
- identical outputs.
282
  * Requires a vLLM build with Qwen3.5 (`qwen3_5`) support, LoRA on `lm_head`, `--logprobs-mode` and
283
  `allowed_token_ids`; tested with a vLLM development build from September 2026. Start-up takes 3–8 minutes
284
  (CUDA-graph capture with LoRA enabled).
285
 
286
- ## What it does
287
 
288
  | kind | question | returns |
289
  |---|---|---|
@@ -304,39 +525,13 @@ D) maintain
304
  ```
305
 
306
  The template is tokenised as one string; the last token's final-norm hidden state goes through a
307
- **linear fp32 head `H → 24 slots`** (`noul` → slots 0–1, `score` → 2–7, `choice` → 8–23). Inactive slots
308
- are masked, a per-kind temperature is applied, and a softmax yields the distribution aligned with your
309
- `options`. One prefill pass — no decoding.
310
-
311
- ## One backbone, two heads
312
-
313
- ```
314
- ┌── lm_head (248,320 × 4096) ─────────────► next-token logits → text generation
315
- Qwen3.5-9B text tower ──┤ (pristine base weights; adapter OFF)
316
- (bit-identical to base)│
317
- └── + LoRA adapter (154 MB) ── decision head (24 × 4096) ─► calibrated typed decision
318
- (adapter ON; head rows were initialised from lm_head rows)
319
- ```
320
-
321
- | mode | weights used | quality |
322
- |---|---|---|
323
- | **Text generation** | backbone + `lm_head`, adapter off | exactly Qwen3.5-9B: HumanEval greedy pass@1 70.7 % (116/164), completions byte-identical to the base model |
324
- | **Typed decisions** | backbone + `adapter/` + `head.safetensors` | the numbers in [Headline results](#headline-results) |
325
-
326
- Why the adapter is shipped unmerged: folding the LoRA into the backbone would let one weight set serve both
327
- heads, but it costs generation quality — the merged backbone with the original `lm_head` scores **61.6 %**
328
- (101/164) on HumanEval, a 9-point drop, even though its perplexity on prose barely moves (3.15 → 3.30).
329
- Keeping the base pristine and applying the 40 M-parameter adapter only on the decision path removes that
330
- trade-off. For decision serving the adapter is merged *in memory* at start-up, so decision latency is the
331
- same as a merged bundle.
332
 
333
- ## Usage
334
 
335
- ### Option A — vLLM (recommended)
336
-
337
- See [Inference with vLLM](#inference-with-vllm-recommended) above.
338
-
339
- ### Option B — plain `transformers` + `peft`
340
 
341
  ```python
342
  import json, torch
@@ -349,13 +544,13 @@ repo = "autotrust/JEV-9B"
349
  tok = AutoTokenizer.from_pretrained(repo)
350
  base = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="cuda") # == Qwen3.5-9B text model
351
 
352
- # --- text generation: the pristine base model, no adapter --------------------------------------
353
  msgs = [{"role": "user", "content": "In two sentences, what is safety stock?"}]
354
  enc = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True).to("cuda")
355
  out = base.generate(**enc, max_new_tokens=80)
356
  print(tok.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True))
357
 
358
- # --- typed decisions: attach the LoRA adapter (merged here for speed) + the 24-slot head --------
359
  model = PeftModel.from_pretrained(base, repo, subfolder="adapter").merge_and_unload()
360
  head = load_file(hf_hub_download(repo, "head.safetensors"))
361
  cfg = json.load(open(hf_hub_download(repo, "judge_config.json")))
@@ -379,69 +574,31 @@ print(decide("noul", "Customer says the parcel arrived damaged and wants their m
379
  # {'false': 0.009, 'true': 0.991}
380
  ```
381
 
382
- `options` are validated: `noul` must be `["false","true"]`, `score` must be `["0".."5"]`, `choice` takes
383
- 2–16 free-text options. To keep both modes in one process, leave the adapter unmerged and run generation
384
- inside `with model.disable_adapter():`.
385
-
386
- ## Why start from a pretrained LLM
387
-
388
- A System One judge is "just a classifier", so one could train a small encoder from scratch or fine-tune a
389
- BERT-class model. autotrust/JEV-9B starts from a pretrained 9 B LLM instead, and that choice — not the 3 GPU-hours of
390
- distillation — is where most of the result above comes from:
391
-
392
- 1. **Pretraining does most of the work; distillation only sharpens.** The 24-slot head is *initialised from
393
- the backbone's own `lm_head` rows* for the verbalizer tokens (`false/true`, `0`–`5`, `A`–`P`), so at
394
- step 0 the head's output is *exactly* the pretrained model's zero-shot restricted next-token distribution
395
- (verified to |Δp| < 1e-5; measured 8.6e-07). Without seeing a single label that starting point already
396
- agrees with TypeSafe Jev 1.13 on 53 % of choice questions and reaches noul AUROC 0.82; distillation takes it to
397
- 90 % / 0.996. World knowledge, instruction reading and long-context handling are inherited, not learned.
398
-
399
- 2. **Extreme sample- and compute-efficiency.** Only **40.2 M parameters** are trained — LoRA r=16 on the
400
- decoder projections plus the 98 k-parameter head, 0.5 % of the 8.0 B text backbone. KL 0.028 was reached after
401
- **0.49 epoch (≈ 320 k rows, ≈ 1.7 GPU-hours on one B200)** and KL 0.021 after 0.93 epoch (≈ 3 GPU-hours);
402
- after the first 64 k rows validation KL was already below 0.10.
403
-
404
- 3. **Transfer to unseen task families.** On the Open-Jev *OOD* split (game-state and workflow-policy
405
- decisions never seen in training) autotrust/JEV-9B reaches KL 0.234 and top-1 0.918 — a pretrained LLM reads the
406
- *content* of a new task instead of matching surface patterns of the 53 training domains.
407
-
408
- 4. **Reads real, long, structured states.** States in the corpus are prose, JSON game states, policy
409
- documents and retrieved passages up to 856 tokens; the backbone supports far longer. BERT-class encoders
410
- stop at 512 tokens and lack the code/JSON pretraining to parse them.
411
-
412
- 5. **Reads options, not positions.** With 30 % option-permutation augmentation during distillation, the
413
- top-1 flip rate under shuffled `choice` options is 3.9 %; the same backbone before distillation flips
414
- 38 % of the time. The pretrained attention binds the letter to the option text almost immediately.
415
-
416
- 6. **Calibration falls out for free.** Distilling *full* teacher distributions with KL (plus an ordinal RPS
417
- term for `score`) on a pretrained backbone gives fitted temperatures of **1.002 / 0.984 / 1.012**: the
418
- model is calibrated to the teacher with no post-hoc correction (ECE 0.0007).
419
-
420
- 7. **The recipe scales without code changes.** The head-initialisation identity holds for any `qwen3_5`
421
- checkpoint without logit soft-capping; the equivalence gate also passes on Qwen3.8-27B (4.5e-07). 9 B was
422
- kept because it already meets the distributional targets.
423
 
424
  ## Evaluation details
425
 
426
- ### Additional metrics (`test_set_30k`, temperature applied)
427
 
428
  | metric | autotrust/JEV-9B |
429
  |---|---|
430
- | `noul` Brier score against the teacher's probability (lower is better) | 0.0015 |
431
  | `score` ranked probability score (lower is better) | 0.0085 |
432
  | Fitted temperatures noul / choice / score | 1.002 / 0.984 / 1.012 |
433
  | Top-1 flip rate when `choice` options are shuffled (1,000 rows × 4 permutations) | 3.9 % |
434
  | Out-of-distribution split — top-1 agreement · `noul` AUROC | 0.918 · 0.989 |
435
  | Throughput — batch of 128 requests on one B200 | 2.5 ms per decision (≈ 400 decisions/s) |
436
- | Throughput — 8 concurrent clients, single requests | 32 requests/s, median 250 ms |
437
 
438
- ### Text-generation head — no-degradation check (HumanEval, greedy pass@1, completion-style prompt)
439
 
440
  | weights | pass@1 | note |
441
  |---|---|---|
442
  | Qwen3.5-9B (base) | 70.7 % (116/164) | same loader and protocol as below |
443
- | **autotrust/JEV-9B — generation path (backbone + `lm_head`, adapter off)** | **70.7 % (116/164)** | all 164 completions byte-identical to the base model |
444
- | LoRA folded into the backbone + base `lm_head` (*not shipped*) | 61.6 % (101/164) | why the adapter is kept separate |
445
 
446
  ### Per source × primitive (`test_set_30k`, temperature applied)
447
 
@@ -450,17 +607,17 @@ distillation — is where most of the result above comes from:
450
  | yuri_v3 — synthetic operational scenarios, labelled by TypeSafe Jev 1.13 | noul | 8,537 | 0.005 | 0.961 | 0.001 | 0.994 | — |
451
  | yuri_v3 | choice | 8,312 | 0.028 | 0.902 | 0.002 | — | — |
452
  | yuri_v3 | score | 8,527 | 0.023 | 0.883 | 0.002 | — | 0.103 |
453
- | openjev_v2 — Open-Jev programmatic tasks | noul | 1,432 | 0.004 | 0.998 | 0.003 | 1.000 | — |
454
  | openjev_v2 | choice | 887 | 0.176 | 0.857 | 0.020 | — | — |
455
  | yuri_v1 — placeholder `[0.5, 0.5]` labels (see Limitations) | noul | 2,260 | 0.000 | — | 0.005 | — | — |
456
 
457
- OOD split (13,058 Open-Jev rows from task families not in training): KL 0.234, top-1 0.918, noul AUROC
458
- 0.989; `choice` KL 0.351 / top-1 0.837 (game-state decisions are the hardest slice).
459
 
460
- Choice option-permutation consistency (1,000 rows × 4 random permutations): mean max |Δp| 0.024, p90 0.055,
461
- top-1 flip rate 3.9 %.
462
 
463
- ### How fast it got there (most recent first)
464
 
465
  Validation KL on a fixed 4 k-row subset; `test_set_30k` metrics after calibration.
466
 
@@ -473,18 +630,19 @@ Validation KL on a fixed 4 k-row subset; `test_set_30k` metrics after calibratio
473
  | step 500 | 64 k | 0.094 | 0.081 | 0.817 | 0.224 | 0.977 | 0.022 |
474
  | untrained backbone with the initialised head (reference point, not the model) | 0 | 0.485 | 0.510 | 0.532 | 1.130 | 0.824 | 0.094 |
475
 
476
- Annealing matters: the v0.7.0 cool-down (500 steps, lr ×0.9 → ×0.02 from step 2000) lowered KL by 25 %;
477
- continuing on the unseen remainder of the epoch with the learning rate decayed to ≈ 0.07× (v0.8.0) lowered it by
478
- another 24 % and added 1.4 points of choice agreement. Checkpoints taken at high learning rate should be annealed.
479
 
480
  ## Training details
481
 
482
  | item | value |
483
  |---|---|
484
- | teacher / data | TypeSafe Jev 1.13 full output distributions (as published in the corpus), `SargeDev/jev-distill-corpus-v3` (740,957 rows; `train` 655,806) |
485
  | backbone | `Qwen/Qwen3.5-9B` text tower only (vision tower and MTP head dropped), bf16, frozen |
486
- | trainable | LoRA r=16, α=32, dropout 0.05 on `in_proj_qkv, in_proj_z, out_proj, q/k/v/o_proj, gate/up/down_proj` (40.1 M, shipped unmerged in `adapter/`) + 24-slot head (98 k, fp32, initialised from `lm_head` rows) |
487
- | loss | KL(teacher ‖ model) over active slots + 0.5 · RPS (ranked probability score) for `score` |
 
488
  | augmentation | 30 % random permutation of `choice` options (targets permuted consistently) |
489
  | batching | 128 rows / step, kind-stratified (≥ 1/6 per primitive), length-bucketed, micro-batches capped at 24 k padded tokens, gradient checkpointing |
490
  | optimiser | AdamW (fused), β=(0.9, 0.98), lr head 2e-4 / LoRA 1e-4, cosine, warmup 3 %, grad-clip 1.0 for 2,500 steps; then continued on the unseen remainder of the epoch (fresh AdamW state, warmup 2 %, lr ×0.9 → cosine) and stopped after 2,250 more steps at lr ≈ ×0.07 — 4,750 steps ≈ 0.93 epoch in total |
@@ -495,30 +653,36 @@ another 24 % and added 1.4 points of choice agreement. Checkpoints taken at high
495
 
496
  ## Limitations
497
 
498
- * **autotrust/JEV-9B mirrors TypeSafe Jev 1.13, including its mistakes.** This is a distillation, not an independent judge: where
499
- the teacher was wrong or uncalibrated, so is autotrust/JEV-9B. Published evaluations of the teacher show it is unreliable
500
- for multi-hop reasoning, arithmetic, dates, counting, and adversarial inputs; autotrust/JEV-9B inherits all of that. It
501
- cannot generate text or explanations *as a decision model*; the generation head is simply the unmodified base
502
- model and knows nothing about the decisions it is packaged with.
503
- * **It inherits the teacher's blind spots, and at 9 B adds some of its own.** On fresh inputs it shoves a poker spot
504
- a solver always checks (0.70, teacher 0.62), 11.5 % of its 16-option answers change with option order alone (teacher
505
- 7.0 %), and it missed a code-rule violation that the 27 B sibling catches. Prefer autotrust/JEV-27B for long option
506
- lists and code-rule checks.
507
- * **Choice agreement is capped by teacher ambiguity.** The teacher's `choice` distributions are soft (median
508
- top-1 probability 0.70). On the 14 % of rows where the teacher's top-2 options are within 0.1 of each other,
509
- argmax agreement is near chance for *any* faithful mimic (0.46 where the gap is < 0.05). On teacher-decisive
510
- rows agreement is 0.954, and autotrust/JEV-9B's argmax captures 97.7 % of the teacher probability mass a perfect mimic
511
- could (0.693 vs 0.709).
512
- * **Fixed option sets.** `noul` and `score` accept only their canonical options; `choice` accepts 2–16
513
- options. Inputs longer than 1,024 tokens are truncated (state only, head 60 % / tail 40 %) at serving unless
514
- you raise the limit.
515
- * **English-centric.** The corpus is English; multilingual behaviour is inherited from the backbone but was not
516
- measured.
517
- * **Placeholder labels in the corpus.** The `yuri_v1` memory-relevance stream is 100 % exact-uniform
518
- `[0.5, 0.5]` — those rows teach nothing about relevance. autotrust/JEV-9B outputs ≈ 0.5 on them by design; do not use it
519
- for memory-relevance scoring without further training.
520
- * **Not for high-stakes decisions.** Use confidence gating: act automatically only above a threshold you
521
- validated on your own data, and route the rest to a stronger model or a human.
 
 
 
 
 
 
522
 
523
  ## Files
524
 
@@ -526,12 +690,12 @@ another 24 % and added 1.4 points of choice agreement. Checkpoints taken at high
526
  model-0000{1..5}-of-00005.safetensors Qwen3.5-9B text backbone incl. lm_head — bit-identical to the base model
527
  (bf16; GDN A_log / gated-norm weights fp32 as in the original), 17.9 GB
528
  model.safetensors.index.json · config.json
529
- adapter/ LoRA decision adapter (peft format, r=16, 40.1 M params, 154 MB) — apply only for decisions
530
  head.safetensors 24-slot decision head (fp32): proj.weight [24, 4096], proj.bias [24]
531
  judge_config.json slot layout, verbalizer token ids, template version, weights_mode=unmerged, provenance
532
  calibration.json per-kind temperatures (+ fit diagnostics)
533
- adapter_vllm/ the same adapter for vLLM: backbone LoRA (zero-padded to r=32) + decision head as an lm_head LoRA,
534
- plus decision_head.json (head bias, verbalizer token ids)
535
  tokenizer.json · tokenizer_config.json · chat_template.jinja
536
  reports/ evaluation reports: test-set evaluation, bundle checks, HumanEval per-problem
537
  results, vLLM measurements, real-world tests, training-milestone reviews
@@ -540,14 +704,14 @@ reports/ evaluation reports: test-set evaluation,
540
  ## License and acknowledgements
541
 
542
  Weights: **Apache-2.0** (base model `Qwen/Qwen3.5-9B` is Apache-2.0; training corpus
543
- `SargeDev/jev-distill-corpus-v3` is Apache-2.0, its `openjev_v2` stream additionally CC0). The System One
544
- framing and the `noul` / `choice` / `score` primitives originate with TypeSafe AI's Jev; autotrust/JEV-9B is an
545
- independent student model trained on public data and shares no weights, code or affiliation with TypeSafe AI.
546
 
547
  ```bibtex
548
- @misc{autotrust_jev_2026,
549
- title = {autotrust/JEV-9B: an open-weights student of TypeSafe Jev 1.13 for calibrated System One decisions (Qwen3.5-9B)},
550
- author = {autotrust},
551
  year = {2026},
552
  url = {https://huggingface.co/autotrust/JEV-9B}
553
  }
 
10
  pipeline_tag: text-classification
11
  tags:
12
  - system-one
13
+ - system-two
14
+ - blocks-of-experts
15
  - typed-decisions
16
  - decision-model
17
  - calibrated-probabilities
 
42
  split: test_set_30k
43
  metrics:
44
  - type: kl_divergence
45
+ name: mean KL(target ‖ model), all test rows (25,376 of 29,955 targets are TypeSafe Jev 1.13 distributions)
46
  value: 0.0210
47
  - type: auroc
48
  name: noul AUROC
49
+ value: 0.996
50
  - type: brier
51
+ name: noul Brier (vs. target probability, all rows)
52
  value: 0.0015
53
  - type: mae
54
  name: score expected-value MAE (0–5 scale)
 
60
  name: choice top-1 agreement (all rows)
61
  value: 0.898
62
  - type: accuracy
63
+ name: choice top-1 agreement (decisive-target rows, top-2 gap ≥ 0.1)
64
  value: 0.954
65
+ - task:
66
+ type: text-generation
67
+ name: code generation — System 2 path (base lm_head, adapter off)
68
+ dataset:
69
+ type: openai/openai_humaneval
70
+ name: HumanEval
71
+ split: test
72
+ metrics:
73
+ - type: pass@1
74
+ name: pass@1 (greedy, completion-style prompt)
75
+ value: 0.707
76
  ---
77
 
78
+ # autotrust/JEV-9B
79
+
80
+ ### AutoTrust's first integrated System 1 + System 2 open model, built with the Blocks of Experts recipe
81
+
82
+ **Fast, calibrated System 1 decisions that are indistinguishable from the closed TypeSafe Jev 1.13 by KL, and
83
+ deliberate System 2 generation and reasoning from an untouched Qwen3.5-9B — one set of weights, one vLLM engine,
84
+ routed per request. The fastest model of the family: it answers a single decision in about a third of the time the
85
+ hosted API takes. Its successor, [autotrust/JEV-27B](https://huggingface.co/autotrust/JEV-27B), is closer to Jev,
86
+ transfers better to unseen tasks and has a stronger System 2.**
87
+
88
+ ## At a glance
89
+
90
+ **Integrated System 1 + System 2, first generation.** JEV-9B is AutoTrust's first open model to serve both modes of
91
+ thinking from a single set of weights; the second generation is
92
+ [autotrust/JEV-27B](https://huggingface.co/autotrust/JEV-27B). *System 1* answers typed questions (`noul` yes/no ·
93
+ `choice` over 2–16 options · `score` on a 0–5 scale) in one forward pass and returns a calibrated probability
94
+ distribution. *System 2* is ordinary text generation with step-by-step reasoning (thinking mode). Both run on the
95
+ same backbone in the same engine, and a request chooses its system.
96
+
97
+ **Blocks of Experts recipe.** Rather than fine-tuning one monolithic model, the Blocks of Experts (BoE) recipe keeps a
98
+ strong pretrained model as a frozen expert block and adds a small, detachable expert block trained for one capability.
99
+ In JEV-9B the System 2 block is Qwen3.5-9B, bit-identical to the release; the System 1 block is 40.2 M trained
100
+ parameters (0.5 % of the backbone), trained in ≈ 3 hours on one B200. Because the blocks stay separate, adding
101
+ System 1 costs System 2 nothing: HumanEval is 70.7 % before and after, with all 164 completions byte-identical. Folding
102
+ the same block into the backbone instead would have cost 9 points (61.6 %).
103
+
104
+ **Indistinguishable from the closed original on System 1, by KL.** On the 25,376 held-out questions (53 domains)
105
+ whose labels are TypeSafe Jev 1.13's own output distributions, the mean KL divergence is **≈ 0.019** (0 = identical).
106
+ An observer who sees sampled decisions gains on average 0.019 nats of evidence per decision about which model produced
107
+ it, so it takes about 54 decisions to gather a single nat. The fidelity extends to the teacher's mistakes (see
108
+ [System 1: indistinguishable from TypeSafe Jev 1.13](#system-1-indistinguishable-from-typesafe-jev-113-by-kl)).
109
+ Among the open Jev reproductions we could find, only the JEV models publish this distribution-level measure
110
+ (see [How JEV-9B compares with other open Jev reproductions](#how-jev-9b-compares-with-other-open-jev-reproductions)).
111
+
112
+ **Faster than the hosted API.** On one B200, a single decision takes a median ≈ 90 ms, against 238–301 ms measured
113
+ independently for the hosted TypeSafe Jev 1.13 API, and one GPU sustains about 15× the decisions per second an
114
+ independent benchmark achieved against that API (see [Speed](#speed-vs-the-hosted-typesafe-jev-113)).
115
+
116
+ **The fast member of the family; JEV-27B is the closer one.** Same recipe, same API: JEV-9B is 2.6× faster than
117
+ JEV-27B on the same benchmark and its weights are a third of the size (18 GB vs 54 GB). JEV-27B lowers mean KL to Jev's
118
+ distributions from ≈ 0.019 to ≈ 0.017, more than halves KL on unseen task families (0.234 → 0.104), keeps 96 % instead
119
+ of 90 % of the teacher's accuracy on an independent 16-option benchmark, and scores 78.0 % instead of 70.7 % on
120
+ HumanEval (see [JEV-9B vs JEV-27B](#jev-9b-vs-jev-27b)).
121
+
122
+ > **Two models, two organisations.** **TypeSafe Jev 1.13** is the hosted, closed-source model made by TypeSafe AI; it
123
+ > is the *teacher* whose published output distributions this model was trained on. **autotrust/JEV-9B** (this
124
+ > repository) is an independent open-weights *student* built by AutoTrust AI from the Apache-2.0 corpus
125
+ > [`SargeDev/jev-distill-corpus-v3`](https://huggingface.co/datasets/SargeDev/jev-distill-corpus-v3). It is not
126
+ > affiliated with, endorsed by, or a product of TypeSafe AI, and shares no weights or code with it.
127
 
128
+ ## Headline results
 
 
 
 
 
 
129
 
130
+ System 1 numbers are on the held-out `test_set_30k` of `jev-distill-corpus-v3`. Its 29,955 rows come from three
131
+ sources: 25,376 rows labelled with TypeSafe Jev 1.13's own output distributions (`yuri_v3`), 2,319 Open-Jev rows with
132
+ programmatic ground-truth labels (`openjev_v2`), and 2,260 placeholder rows (`yuri_v1`). Rows marked *Jev-labelled* use
133
+ only the first group.
 
 
 
134
 
135
+ | | What is measured | autotrust/JEV-9B | How to read it |
136
+ |---|---|---|---|
137
+ | **System 1** | Mean KL divergence from TypeSafe Jev 1.13's distributions, Jev-labelled rows, 0 = identical | **≈ 0.019** | Indistinguishable from the teacher's decisions at this resolution: ≈ 54 sampled decisions to gather one nat of evidence |
138
+ | | Mean KL to all test targets (Jev, programmatic and placeholder labels) | **0.021** | The figure in the model index above |
139
+ | | Yes/no AUROC (`noul`), Jev-labelled rows | **0.994** | Ranks true vs. false almost perfectly (0.996 over all rows) |
140
+ | | Choice top-1 agreement with Jev, Jev-labelled rows | **90.2 %** | 95.4 % over all rows where the target's top two options differ by ≥ 0.1; on near ties any faithful copy agrees about half the time |
141
+ | | Rating error (`score`, 0–5 scale), mean absolute error of the expected rating | **0.103** | About one tenth of a rating step |
142
+ | | Expected calibration error | **0.0007** | A stated 80 % is an 80 %; fitted temperatures ≈ 1.00, no post-hoc correction needed |
143
+ | | KL to the programmatic labels of task families never seen in training (Open-Jev OOD split) | **0.234** | Transfer to new tasks; these labels are ground truth, not Jev's outputs. JEV-27B: 0.104 |
144
+ | | Independent benchmark with human gold labels, 16 options | **90 % of the teacher** (0.694 vs 0.769) | 94–97 % of the teacher at 2, 4 and 8 options; see [Benchmark highlights](#benchmark-highlights) |
145
+ | **System 2** | HumanEval pass@1, greedy | **70.7 %** | Identical to Qwen3.5-9B (116/164); all 164 completions byte-identical to the base model |
146
+ | **Speed** | Single decision, median, one B200 | **≈ 90 ms** | Hosted TypeSafe Jev 1.13, measured independently: 238 ms mean, 291–301 ms median |
147
+ | | Decisions per second on the independent benchmark, one B200 | **≈ 340** | ≈ 15× the 23 per second measured against the hosted API; see [Speed](#speed-vs-the-hosted-typesafe-jev-113) |
148
+ | | Batched, 128 decisions per batch | **2.5 ms** per decision | With vLLM: 205 decisions/s over HTTP at 256 concurrent clients, text generation ≈ 50× faster than the PyTorch path |
149
+ | **Efficiency** | Trained parameters | **40.2 M** (0.5 % of 7.9 B) | ≈ 3 B200-hours, 0.93 epoch ��� 608 k rows |
150
+
151
+ ## JEV-9B vs JEV-27B
152
+
153
+ JEV-9B was AutoTrust's first integrated System 1 + System 2 model.
154
+ [autotrust/JEV-27B](https://huggingface.co/autotrust/JEV-27B), the second generation, uses the same recipe, code,
155
+ hyper-parameters, API and two-block packaging; only the backbone and memory settings changed. Both are evaluated on the
156
+ same held-out test set and the same independent benchmark.
157
+
158
+ <p align="center">
159
+ <a href="https://huggingface.co/autotrust/JEV-27B/blob/main/27b-2.jpg"><img src="https://huggingface.co/autotrust/JEV-27B/resolve/main/27b-2.jpg" alt="JEV family benchmark highlights: KL to TypeSafe Jev 1.13 by question type, accuracy as a percentage of Jev on an independent benchmark, HumanEval for the System 2 path, and speed against the hosted API, for JEV-9B (light bars) and JEV-27B (dark bars)" width="100%"></a>
160
+ <br>
161
+ <sub><b>JEV family benchmark highlights</b> (chart from the JEV-27B repository; light bars = JEV-9B). A · KL to Jev by question type · B · accuracy as % of Jev on an independent benchmark · C · System 2 unchanged · D · speed vs the hosted API · click to enlarge</sub>
162
+ </p>
163
+
164
+ | | **JEV-9B** | JEV-27B | JEV-27B vs JEV-9B |
165
+ |---|---|---|---|
166
+ | Backbone | Qwen3.5-9B | Qwen3.8-27B | |
167
+ | **System 1** — mean KL to TypeSafe Jev 1.13, Jev-labelled rows | ≈ 0.019 | **≈ 0.017** | ≈ −11 % |
168
+ | Mean KL to all test targets | 0.021 | **0.019** | −11 % |
169
+ | KL to ground-truth labels, unseen task families (OOD) | 0.234 | **0.104** | −56 % |
170
+ | Top-1 accuracy, unseen task families (OOD) | 0.918 | **0.942** | +2.4 pts |
171
+ | Choice top-1 agreement with Jev, Jev-labelled rows | 90.2 % | **90.5 %** | +0.3 pts |
172
+ | Rating error (`score` MAE, all Jev-labelled) | 0.103 | **0.098** | −5 % |
173
+ | Top-1 flips under option shuffle (test set) | 3.9 % | **2.9 %** | −1.0 pt |
174
+ | Yes/no AUROC (`noul`), Jev-labelled rows | 0.994 | **0.995** | +0.001 |
175
+ | Calibration error (ECE) | **0.0007** | 0.0009 | JEV-9B slightly lower; both below 0.001 |
176
+ | Independent benchmark, 16 options — % of teacher accuracy | 90 % | **96 %** | +6 pts |
177
+ | Independent benchmark — answers changed by option order alone (teacher: 7.0 %) | 11.5 % | **7.4 %** | JEV-27B is close to the teacher's 7.0 % |
178
+ | **System 2** — HumanEval pass@1 (greedy) | 70.7 % | **78.0 %** | +7.3 pts |
179
+ | Latency on one B200 — single request / batched | **≈ 90 ms / 2.5 ms** | 137 ms / 4.2 ms | JEV-9B is faster |
180
+ | Benchmark throughput — 14,400 decisions on one B200 | **42 s** | 110 s | JEV-9B is 2.6× faster |
181
+ | Download size (backbone + adapter) | **18 GB** | 54 GB | |
182
+ | Trained parameters / compute | 40.2 M / ≈ 3 B200-hours | 108.9 M / ≈ 9.2 B200-hours | |
183
+
184
+ On the fresh Hacker News, V2EX and community examples (illustrations, not a benchmark), JEV-9B got 92 of 96 decisions
185
+ right against 95 of 96 for JEV-27B. The difference is on the harder tasks: JEV-9B misses a TypeScript port that
186
+ breaks a "branded, range-checked integer" rule (0.33; JEV-27B 0.93) and flags a CEO wire-transfer fraud message with
187
+ less confidence (0.56; JEV-27B 0.84).
188
+
189
+ **Which to pick.** For routing, moderation, topic triage and short option lists, JEV-9B gives nearly the same answers
190
+ 2.6× faster (14,400 benchmark decisions in 42 s vs 110 s on one B200) with a third of the weight memory. For long option lists
191
+ (more than about 8), unfamiliar task families, code-rule checks, fraud screening, or when the System 2 path matters, use
192
+ [autotrust/JEV-27B](https://huggingface.co/autotrust/JEV-27B).
193
 
194
+ ## How JEV-9B compares with other open Jev reproductions
195
 
196
+ Dozens of open reproductions of TypeSafe Jev appeared within weeks of its launch; the community
197
+ [Jev Decision Index](https://huggingface.co/spaces/multimodalart/jev-decision-index) (formerly the Jev Reproductions
198
+ Tracker) evaluates 55 of them. Most are trained on human or programmatic gold labels, or on their own synthetic data, so
199
+ they aim to match or beat Jev's accuracy rather than reproduce its probabilities. "Closest to Jev" therefore depends on
200
+ how closeness is measured:
201
 
202
+ | measure of closeness to TypeSafe Jev 1.13 | published results (snapshot of 25 September 2026) | where JEV-9B stands |
203
  |---|---|---|
204
+ | **Distribution level:** KL to Jev's own output distributions on held-out rows | JEV-27B ≈ 0.017 and JEV-9B ≈ 0.019 on 25,376 Jev-labelled rows. We found no other open reproduction that publishes this measure. | Second lowest published, after JEV-27B |
205
+ | **Accuracy relative to Jev** on [`decision-models-under-pressure`](https://github.com/gazelle93/decision-models-under-pressure), 16 options, human gold labels | JEV-27B 96 % · JEV-9B 90 % · Laya 90 % · DeBERTa-v3-large zero-shot 90 % · DeBERTa-v3-base zero-shot 83 % · GLiClass-large 81 % · bge-large 73 % · gte-large 69 % | Level with the best of the other models measured there; JEV-27B is closer (JEV rows are AutoTrust re-runs of the same items; the others were run by the benchmark's author) |
206
+ | **Score parity on community leaderboards** | [Decision Index 0.2](https://huggingface.co/spaces/multimodalart/jev-decision-index): Jev 51.67, AutoJev-27B 50.94. [JevBench v1.4.2](https://github.com/fstandhartinger/jevbench): decider-4b v2 64.13, Jev 63.29, JevK5 62.04. [Open-Jev](https://zefan-cai.github.io/open-jev/benchmarks/) public JevBench subset: Jev 200/231, Open-Jev 27B v1.1 197/231 | Not yet evaluated |
207
+
208
+ On the evidence published today, the two JEV models are the closest open models to TypeSafe Jev 1.13 at the level of
209
+ output distributions, with JEV-9B second to JEV-27B. On the independent benchmark JEV-9B is level with the best of the
210
+ other models measured there, not ahead of them. It has not yet been run on the Decision Index or JevBench, where
211
+ AutoJev-27B scores within about one point of Jev and decider-4b v2 edges ahead of it, so we do not claim it is the
212
+ closest by every measure. Note that some reproductions report beating Jev on their own test sets (AutoJev-27B reports
213
+ 84.60 % against Jev's 82.79 %); that is a different goal from reproducing Jev's behaviour.
214
 
215
+ *Not to be confused with AutoJev-27B (`denis-pplx/autojev-27b`), an unrelated Qwen3.8-27B decision model trained with
216
+ full-weight SFT on its own data.*
 
217
 
218
+ ## Speed vs the hosted TypeSafe Jev 1.13
219
 
220
+ TypeSafe does not publish Jev's size or hardware; it reports 70–500 ms end to end. Independent measurements, and ours:
221
+
222
+ | | TypeSafe Jev 1.13, hosted API | **JEV-9B, one B200** | JEV-27B, one B200 |
223
+ |---|---|---|---|
224
+ | One decision, single request | 238 ms mean over 29,600 calls ([`decision-models-under-pressure`](https://github.com/gazelle93/decision-models-under-pressure)); 291–301 ms median on three workloads ([Open-Jev](https://zefan-cai.github.io/open-jev/benchmarks/)) | **≈ 90 ms** median (87 ms) | 137 ms median |
225
+ | Decisions per second on `decision-models-under-pressure` | 23, with 5 client workers and one question per call | **≈ 340** (14,400 in 42 s) | ≈ 130 (14,400 in 110 s) |
226
+ | Batched, 128 decisions per batch | — | **2.5 ms** per decision | 4.2 ms per decision |
227
+
228
+ So JEV-9B answers a single decision in roughly a third of the time (JEV-27B in roughly half), and one GPU sustains
229
+ about 15× (JEV-27B: about 6×) the throughput the benchmark's author achieved against the hosted API. Read these with
230
+ the caveats: our latencies are measured on the serving host with no network hop, while the hosted numbers include
231
+ internet, TLS and queueing; hosted throughput depends on client concurrency and the API's rate limits; Jev's latency is
232
+ roughly flat in the number of questions per request, so bundling questions narrows the throughput gap; and our figures
233
+ are self-reported while Jev's come from third parties. The two throughput runs use the same benchmark but not an
234
+ identical call set (ours stops at 16 options).
235
+
236
+ ## System 1: indistinguishable from TypeSafe Jev 1.13, by KL
237
+
238
+ **What the number means.** KL(Jev ‖ model) is the expected log-likelihood ratio, per sampled decision, between
239
+ TypeSafe Jev 1.13 and the student when the decision comes from Jev. On the 25,376 held-out rows whose targets are
240
+ Jev's own output distributions, the mean is ≈ 0.019 nats (computed from the per-slice values below, which are
241
+ published to three decimals): one decision carries almost no evidence about which of the two models produced it, and
242
+ an observer needs about 1 / KL ≈ 54 independent decisions to accumulate one nat (a likelihood ratio of about e ≈ 2.7 : 1).
243
+
244
+ For scale, Jev is not deterministic itself: an independent study found it changes its answer on 4.3 % of repeated,
245
+ identical 64-option calls, and it returns probabilities rounded to two decimals, which is the resolution of the
246
+ targets used here.
247
+
248
+ | Jev-labelled slice (`yuri_v3`, `test_set_30k`) | n | KL | ≈ decisions to gather one nat (1 / KL) |
249
+ |---|---|---|---|
250
+ | `noul` | 8,537 | 0.005 | ≈ 200 |
251
+ | `choice` | 8,312 | 0.028 | ≈ 36 |
252
+ | `score` | 8,527 | 0.023 | ≈ 43 |
253
+ | **all Jev-labelled rows** | **25,376** | **≈ 0.019** | **≈ 54** |
254
 
255
+ JEV-27B reaches ≈ 0.017 (≈ 60 decisions per nat) on the same rows.
256
+
257
+ The other test rows are not labelled by Jev and are not part of this claim: Open-Jev rows carry programmatic ground
258
+ truth (in-distribution KL 0.004 for `noul`, 0.176 for `choice`; 0.234 on the OOD split of unseen task families), and
259
+ the `yuri_v1` rows carry placeholder labels. No Jev-labelled out-of-distribution set exists in the corpus, so the claim
260
+ is established on the 53 training domains; outside them, the independent benchmark with human labels (90–97 % of Jev's
261
+ accuracy) is the best available evidence.
262
+
263
+ **Fidelity includes the teacher's mistakes.** On a poker spot where a solver always checks, TypeSafe Jev 1.13 shoves
264
+ with 0.62 in a published test; JEV-9B shoves too, with 0.70 (JEV-27B 0.63). A faithful copy of System 1 is also a
265
+ faithful copy of its blind spots. At 9 B the student also adds some of its own: on an independent benchmark 11.5 % of
266
+ its 16-option answers change when only the option order changes, against 7.0 % for the teacher (JEV-27B 7.4 %).
267
+
268
+ ## The Blocks of Experts recipe
269
+
270
+ ```
271
+ ┌── System 2 block: lm_head (248,320 × 4096) ───────► text generation and reasoning
272
+ Request ─► Router ─► Qwen3.5-9B backbone (frozen, bit-identical to the base) (adapter off; HumanEval 70.7 % = base)
273
+ per │
274
+ request └── + System 1 block: LoRA (40.1 M) + 24-slot head (98 k) ─► calibrated typed decision
275
+ (adapter on, decision path only) (one prefill pass; KL ≈ 0.019 to Jev)
276
+ ```
277
+
278
+ | block | what it is | parameters | trained? | used for |
279
+ |---|---|---|---|---|
280
+ | Backbone | `Qwen/Qwen3.5-9B` text tower (vision tower and MTP head dropped), bf16 | 7.9 B | no — bit-identical to the base | both systems |
281
+ | **System 2 block** | the original `lm_head` (248,320 × 4096) | part of the base | no | text generation and step-by-step reasoning |
282
+ | **System 1 block** | LoRA r=16 on the decoder projections + a 24-slot fp32 decision head initialised from `lm_head` rows | 40.1 M + 98 k | yes, ≈ 3 B200-hours | calibrated typed decisions |
283
+ | Router | per request: the vLLM LoRA module `jev-decision`, or `peft` adapter on/off | — | — | chooses the system |
284
+
285
+ **Why separate blocks rather than one merged fine-tune.** Folding the System 1 LoRA into the backbone would let a
286
+ single weight set serve both heads, but it costs generation quality: the merged backbone with the original `lm_head`
287
+ scores 61.6 % (101/164) on HumanEval against 70.7 % for the base, a 9-point drop, even though prose perplexity barely
288
+ moves (3.15 → 3.30). Keeping the backbone pristine and applying the System 1 block only on the decision path removes
289
+ that trade-off. For decision serving the adapter is merged *in memory* at start-up, so decision latency matches a
290
+ merged bundle.
291
+
292
+ **Why the recipe is this efficient.**
293
+
294
+ 1. **Pretraining does most of the work; distillation sharpens.** The decision head is initialised from the backbone's
295
+ own `lm_head` rows for the verbalizer tokens (`false/true`, `0`–`5`, `A`–`P`), so at step 0 its output equals the
296
+ pretrained model's zero-shot restricted next-token distribution (verified to |Δp| < 1e-5; measured 8.6e-07). Before
297
+ seeing a single label it already agrees with the test targets on 53 % of `choice` questions with `noul` AUROC 0.82;
298
+ distillation takes it to 90 % / 0.996.
299
+ 2. **Small trainable footprint.** 40.2 M parameters — 0.5 % of the backbone. Validation KL was already below 0.10 after
300
+ the first 64 k rows, test KL reached 0.028 after 0.49 epoch (≈ 1.7 B200-hours) and 0.021 after 0.93 epoch.
301
+ 3. **Transfer to unseen tasks.** The pretrained backbone reads the *content* of a new task instead of matching surface
302
+ patterns of the training domains: KL 0.234 and top-1 0.918 against the programmatic labels of the OOD split. It
303
+ also reads real, long, structured states (prose, JSON game states, policy documents; up to 856 tokens in the corpus).
304
+ 4. **Reads options, not positions.** With 30 % option-permutation augmentation, the top-1 flip rate under shuffled
305
+ `choice` options is 3.9 %; the same backbone before distillation flips 38 % of the time.
306
+ 5. **Calibration falls out of the objective.** Distilling full teacher distributions with KL (plus an ordinal RPS term
307
+ for `score`) gives fitted temperatures of 1.002 / 0.984 / 1.012 and ECE 0.0007 with no post-hoc correction.
308
+ 6. **It scales without code changes, and scale pays off.** The same code, hyper-parameters and packaging produced the
309
+ second-generation JEV-27B; only `model_path` and memory settings changed (the head-initialisation identity holds
310
+ there too, 4.5e-07). Going from 9 B to 27 B lowers KL to Jev from ≈ 0.019 to ≈ 0.017, halves OOD KL
311
+ (0.234 → 0.104), and raises the System 2 path from 70.7 % to 78.0 % on HumanEval.
312
+
313
+ ## Benchmark highlights
314
 
315
  ### Independent benchmark: side by side with TypeSafe Jev 1.13
316
 
317
  [`gazelle93/decision-models-under-pressure`](https://github.com/gazelle93/decision-models-under-pressure) (published
318
  25 Sep 2026) asks decision models to pick the right label for real texts from CLINC-150, MTOP, GoEmotions, DBpedia and
319
+ financial tweets under three kinds of pressure: more options, near-miss options, and shuffled option order. The labels
320
+ are human gold labels, none of this data is in our training set, and TypeSafe Jev 1.13's results are published with it.
321
+ We re-ran the same items with autotrust/JEV-9B and autotrust/JEV-27B, up to our 16-option limit.
322
 
323
  | | TypeSafe Jev 1.13 (published) | **autotrust/JEV-9B** | autotrust/JEV-27B |
324
  |---|---|---|---|
 
328
  | Answers changed by shuffling the options alone (16 options, 5 orderings) | 7.0 % | **11.5 %** | 7.4 % |
329
  | Time for 14,400 decisions on one B200 | — | **42 s** | 110 s |
330
 
331
+ On data it was never trained on, JEV-9B reaches 97 % of the teacher's accuracy with 2 and 4 options, 94 % with 8 and
332
+ 90 % with 16: it falls behind faster than JEV-27B (96–98 %) as the option list grows, loses a little more on near-miss
333
+ options, and is more sensitive to option order than the teacher. Our run follows the benchmark's published method (gold
334
+ plus the first K−1 distractors of a pool, shuffled per item); the orderings are seeded differently, so compare
335
+ aggregates, not individual items.
 
336
 
337
  ### Fresh examples (Hacker News and V2EX, 23–25 September 2026)
338
 
339
+ Expected answers were written by hand before the model was run. These are illustrations (≈ 110 decisions), not a
340
  benchmark.
341
 
342
  | task | autotrust/JEV-9B | autotrust/JEV-27B |
 
350
  |---|---|---|
351
  | HN: "Two-tier encryption in the UK" | topic (10 options) | security & privacy · 0.87 |
352
  | HN: "Using LLMs to trace alchemical knowledge and decode 17th century letters" | about AI? | P(true) = 0.88 |
 
353
  | HN comment: "Please stop this. We've asked you before to observe the guidelines…" | what is it mainly doing? | moderating the discussion · 0.78 |
354
  | V2EX: "一个不需要 gemini pro 的完全免费��注册 Muse 的方法 … 邀请码:…" | contains a referral / invite code? | P(true) = 1.00 |
355
  | V2EX: "今天中秋节,还要加班的有吗?来报道下" | promotes a product or paid offer? | P(true) = 0.01 |
356
  | Diff replacing a parameterised query with `"… WHERE id = " + request.args["id"]` | introduces a security vulnerability? | P(true) = 0.93 (0.17 for a variable rename) |
357
  | "I'm not happy with the fit. What are my options here?" | asking for a refund? | P(true) = 0.17 (TypeSafe's docs report 0.22 for Jev on this exact text) |
358
 
359
+ Where it failed or wavered:
360
 
361
+ * **Code-rule check**: missed a TypeScript file that declares `const port: number = Number(process.env.PORT)` against
362
+ the rule "a port must be a branded, range-checked integer" (0.33); JEV-27B flags it (0.93).
363
+ * **Fraud screening**: a CEO wire-transfer (business-email-compromise) message is flagged, but only at 0.56 (JEV-27B 0.84).
364
  * **Comment intent**: "Because it's not a real argument. It's a deflection people use." read as attacking another
365
  commenter (0.66) rather than arguing a point.
366
+ * **Chinese promotion**: a V2EX post launching a paid HTTPS debugging tool was not flagged as promotional (0.44;
367
+ JEV-27B 0.77).
368
+ * **A poker spot with the nuts** (check or shove four times the pot; a solver checks 100 %): shoves with 0.70; the
369
+ teacher shoved with 0.62, so this mistake comes from the teacher.
370
+ * Counting ("more than 3 fruits?" / "more than 5?" for a list of 4: 0.84 / 0.37), date comparisons and an instruction
371
+ injected inside the state were handled correctly, but on a handful of examples only.
372
 
373
  Per-example outputs and the benchmark aggregates are in `reports/realworld_9b.json` (the HN and V2EX inputs came from
374
  their public APIs on 25 September 2026).
375
 
376
+ ## Quickstart with vLLM (recommended)
377
 
378
+ **One vLLM engine serves both systems from the same pristine weights.** Ordinary requests go through the base
379
+ `lm_head` (System 2, exactly Qwen3.5-9B); requests addressed to the LoRA module `jev-decision` go through the decision
380
+ head (System 1). `adapter_vllm/` contains the backbone LoRA plus the 24-slot decision head re-expressed as an `lm_head`
381
+ LoRA (only the 24 verbalizer rows change), so a typed decision is a single prefill step with `max_tokens=1`,
382
+ constrained to the option tokens and read back as log-probabilities.
383
 
384
  ### 1 — Start the server (OpenAI-compatible)
385
 
386
  ```bash
387
+ hf download autotrust/JEV-9B --local-dir JEV-9B # ~18 GB
388
  vllm serve JEV-9B --served-model-name autotrust/JEV-9B \
389
  --enable-lora --max-lora-rank 32 --lora-modules jev-decision=JEV-9B/adapter_vllm \
390
  --logprobs-mode processed_logprobs --max-model-len 4096
391
  ```
392
 
393
+ `--logprobs-mode processed_logprobs` is required: it makes the returned log-probabilities respect `allowed_token_ids`.
394
+ `--max-model-len 4096` is sized for decisions; raise it (for example to 16384) if System 2 requests will think at
395
+ length. Add `--enable-prefix-caching --mamba-cache-mode align` if you ask many questions about the same state.
396
 
397
+ ### 2 — System 2: generation and reasoning (the unmodified base model)
398
 
399
  ```bash
400
  curl -s localhost:8000/v1/chat/completions -H 'content-type: application/json' -d '{
 
403
  "max_tokens": 60, "chat_template_kwargs": {"enable_thinking": false}}'
404
  ```
405
 
406
+ Set `"enable_thinking": true` for deliberate, step-by-step reasoning. This path is Qwen3.5-9B unchanged; see the
407
+ [Qwen3.5-9B model card](https://huggingface.co/Qwen/Qwen3.5-9B) for its reasoning benchmarks and recommended sampling
408
+ settings.
409
+
410
+ ### 3 — System 1: typed decisions (Python, only `requests` + two small JSON files)
411
 
412
  ```python
413
  import json, math, requests
 
439
  "Is the customer asking for a refund?"))
440
  ```
441
 
442
+ Values can differ in the third decimal between runs: vLLM computes in bf16 and results depend slightly on which
443
+ requests are batched together. The head bias and the temperature are applied client-side; the log-softmax normaliser
444
  that vLLM applies cancels out, so the result is exactly the decision head's calibrated distribution.
445
 
446
+ ### 4 — System 1 → System 2: confidence-gated escalation
447
+
448
+ Because both systems live in one engine, a common pattern is to let System 1 answer when it is confident and hand the
449
+ rest to System 2. This is a usage pattern, not a configuration we have benchmarked; pick the threshold on your own
450
+ validation data, and serve with a `--max-model-len` large enough for the reasoning budget.
451
+
452
+ ```python
453
+ def solve(state, question, options, threshold=0.90):
454
+ p = decide("choice", state, question, options) # System 1: one prefill pass
455
+ best = max(p, key=p.get)
456
+ if p[best] >= threshold:
457
+ return {"system": 1, "answer": best, "distribution": p}
458
+ prompt = (f"{state}\n\nQuestion: {question}\nOptions: " + "; ".join(options)
459
+ + "\nThink it through, then give exactly one option on the last line.")
460
+ r = requests.post(f"{URL}/v1/chat/completions", json={ # System 2: same engine, base lm_head
461
+ "model": "autotrust/JEV-9B",
462
+ "messages": [{"role": "user", "content": prompt}],
463
+ "max_tokens": 8192, "chat_template_kwargs": {"enable_thinking": True}}).json()
464
+ return {"system": 2, "reply": r["choices"][0]["message"]["content"], "system1_distribution": p}
465
+ ```
466
+
467
  ### Offline / batch (Python API)
468
 
469
  ```python
 
473
  llm = LLM("JEV-9B", enable_lora=True, max_lora_rank=32, logprobs_mode="processed_logprobs", max_model_len=4096)
474
  decision = LoRARequest("jev-decision", 1, "JEV-9B/adapter_vllm")
475
 
476
+ gen = llm.generate(["..."], SamplingParams(temperature=0.0, max_tokens=256)) # System 2, no LoRA
477
+ dec = llm.generate([prompt], [SamplingParams(max_tokens=1, temperature=1.0, # System 1
478
  allowed_token_ids=ids, logprobs=len(ids))],
479
  lora_request=decision) # then + bias, / T as above
480
  ```
481
 
482
+ Mixed batches work too: pass a per-request `lora_request` list (`None` for System 2, `decision` for System 1) and both
483
+ systems are served in the same `generate` call.
484
 
485
  ### Measured on one B200
486
 
487
  | workload | PyTorch path | **vLLM** |
488
  |---|---|---|
489
+ | System 2 — 164 HumanEval completions (greedy, ≤ 384 new tokens) | 165 s | **3.3 s** (≈ 50×) |
490
+ | System 1 — offline batch, 29,955 test questions | 75 s (398 q/s) | 80 s (374 q/s) |
491
+ | System 1 over HTTP — 64 / 256 concurrent clients | — | 150 / 205 req/s |
492
+ | System 1 fidelity vs. the PyTorch path | test KL 0.0210 | test KL 0.0211; mean \|Δp\| 0.0008 over HTTP |
493
  | Many questions about one state, `--enable-prefix-caching` | — | +14–20 % throughput |
494
 
495
  Notes:
496
+ * The big win is on System 2: generation is ≈ 50× faster. A decision is a single prefill pass with no decoding, so at
497
+ 9 B offline batch throughput is about the same as the PyTorch path (at 27 B vLLM is 1.7× faster); for System 1 vLLM
498
+ mainly buys serving: continuous batching under concurrency, an OpenAI-compatible API, and one engine for both systems.
499
+ * Prefix caching: this architecture mixes Gated DeltaNet and attention layers, and vLLM caches it in blocks of 528
500
+ tokens, so only shared prefixes longer than 528 tokens are reused. The template puts `[kind]` before `[state]`, so
501
+ only questions of the same kind share a prefix. On 293 real states × 7.7 yes/no questions each (≈ 480-token states),
502
+ prefix caching served 19.6 % of prompt tokens from cache (+14–20 % throughput) with identical outputs.
 
503
  * Requires a vLLM build with Qwen3.5 (`qwen3_5`) support, LoRA on `lm_head`, `--logprobs-mode` and
504
  `allowed_token_ids`; tested with a vLLM development build from September 2026. Start-up takes 3–8 minutes
505
  (CUDA-graph capture with LoRA enabled).
506
 
507
+ ## What System 1 does
508
 
509
  | kind | question | returns |
510
  |---|---|---|
 
525
  ```
526
 
527
  The template is tokenised as one string; the last token's final-norm hidden state goes through a
528
+ **linear fp32 head `H → 24 slots`** (`noul` → slots 0–1, `score` → 2–7, `choice` → 8–23). Inactive slots are masked,
529
+ a per-kind temperature is applied, and a softmax yields the distribution aligned with your `options`. One prefill
530
+ pass, no decoding.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
531
 
532
+ ## Other ways to run it
533
 
534
+ ### Plain `transformers` + `peft`
 
 
 
 
535
 
536
  ```python
537
  import json, torch
 
544
  tok = AutoTokenizer.from_pretrained(repo)
545
  base = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="cuda") # == Qwen3.5-9B text model
546
 
547
+ # --- System 2: the pristine base model, no adapter ----------------------------------------------
548
  msgs = [{"role": "user", "content": "In two sentences, what is safety stock?"}]
549
  enc = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True).to("cuda")
550
  out = base.generate(**enc, max_new_tokens=80)
551
  print(tok.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True))
552
 
553
+ # --- System 1: attach the LoRA adapter (merged here for speed) + the 24-slot head ---------------
554
  model = PeftModel.from_pretrained(base, repo, subfolder="adapter").merge_and_unload()
555
  head = load_file(hf_hub_download(repo, "head.safetensors"))
556
  cfg = json.load(open(hf_hub_download(repo, "judge_config.json")))
 
574
  # {'false': 0.009, 'true': 0.991}
575
  ```
576
 
577
+ `options` are validated: `noul` must be `["false","true"]`, `score` must be `["0".."5"]`, `choice` takes 2–16
578
+ free-text options. Note that `merge_and_unload()` above changes the backbone for the rest of the process; to keep both
579
+ systems in one process, leave the adapter unmerged and run System 2 inside `with model.disable_adapter():`.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
580
 
581
  ## Evaluation details
582
 
583
+ ### Additional System 1 metrics (`test_set_30k`, temperature applied)
584
 
585
  | metric | autotrust/JEV-9B |
586
  |---|---|
587
+ | `noul` Brier score against the target probability, all rows (lower is better) | 0.0015 |
588
  | `score` ranked probability score (lower is better) | 0.0085 |
589
  | Fitted temperatures noul / choice / score | 1.002 / 0.984 / 1.012 |
590
  | Top-1 flip rate when `choice` options are shuffled (1,000 rows × 4 permutations) | 3.9 % |
591
  | Out-of-distribution split — top-1 agreement · `noul` AUROC | 0.918 · 0.989 |
592
  | Throughput — batch of 128 requests on one B200 | 2.5 ms per decision (≈ 400 decisions/s) |
593
+ | Single request on one B200 (median) | 87 ms |
594
 
595
+ ### System 2 — no-degradation check (HumanEval, greedy pass@1, completion-style prompt)
596
 
597
  | weights | pass@1 | note |
598
  |---|---|---|
599
  | Qwen3.5-9B (base) | 70.7 % (116/164) | same loader and protocol as below |
600
+ | **autotrust/JEV-9B — System 2 path (backbone + `lm_head`, adapter off)** | **70.7 % (116/164)** | all 164 completions byte-identical to the base model |
601
+ | System 1 LoRA folded into the backbone + base `lm_head` (*not shipped*) | 61.6 % (101/164) | why the blocks are kept separate |
602
 
603
  ### Per source × primitive (`test_set_30k`, temperature applied)
604
 
 
607
  | yuri_v3 — synthetic operational scenarios, labelled by TypeSafe Jev 1.13 | noul | 8,537 | 0.005 | 0.961 | 0.001 | 0.994 | — |
608
  | yuri_v3 | choice | 8,312 | 0.028 | 0.902 | 0.002 | — | — |
609
  | yuri_v3 | score | 8,527 | 0.023 | 0.883 | 0.002 | — | 0.103 |
610
+ | openjev_v2 — Open-Jev programmatic tasks, ground-truth labels (not Jev) | noul | 1,432 | 0.004 | 0.998 | 0.003 | 1.000 | — |
611
  | openjev_v2 | choice | 887 | 0.176 | 0.857 | 0.020 | — | — |
612
  | yuri_v1 — placeholder `[0.5, 0.5]` labels (see Limitations) | noul | 2,260 | 0.000 | — | 0.005 | — | — |
613
 
614
+ OOD split (13,058 Open-Jev rows from task families not in training, programmatic labels): KL 0.234, top-1 0.918, noul
615
+ AUROC 0.989; `choice` KL 0.351 / top-1 0.837 (game-state decisions are the hardest slice).
616
 
617
+ Choice option-permutation consistency (1,000 rows × 4 random permutations): mean max |Δp| 0.024, p90 0.055, top-1
618
+ flip rate 3.9 %.
619
 
620
+ ### Training trajectory (most recent first)
621
 
622
  Validation KL on a fixed 4 k-row subset; `test_set_30k` metrics after calibration.
623
 
 
630
  | step 500 | 64 k | 0.094 | 0.081 | 0.817 | 0.224 | 0.977 | 0.022 |
631
  | untrained backbone with the initialised head (reference point, not the model) | 0 | 0.485 | 0.510 | 0.532 | 1.130 | 0.824 | 0.094 |
632
 
633
+ Annealing matters: the v0.7.0 cool-down (500 steps, lr ×0.9 → ×0.02 from step 2000) lowered KL by 25 %; continuing on
634
+ the unseen remainder of the epoch with the learning rate decayed to ≈ 0.07× (v0.8.0) lowered it by another 24 % and
635
+ added 1.4 points of choice agreement. JEV-27B folds this into a single cosine schedule.
636
 
637
  ## Training details
638
 
639
  | item | value |
640
  |---|---|
641
+ | teacher / data | `SargeDev/jev-distill-corpus-v3` (740,957 rows; `train` 655,806) with three streams: `yuri_v3` (498,010 rows, TypeSafe Jev 1.13 full output distributions via OpenRouter), `openjev_v2` (94,801 rows, Open-Jev programmatic labels, CC0), `yuri_v1` (148,154 rows, placeholder labels, down-weighted) |
642
  | backbone | `Qwen/Qwen3.5-9B` text tower only (vision tower and MTP head dropped), bf16, frozen |
643
+ | System 1 block (trainable) | LoRA r=16, α=32, dropout 0.05 on `in_proj_qkv, in_proj_z, out_proj, q/k/v/o_proj, gate/up/down_proj` (40.1 M, shipped unmerged in `adapter/`) + 24-slot head (98 k, fp32, initialised from `lm_head` rows) |
644
+ | System 2 block | the original `lm_head`, not trained |
645
+ | loss | KL(target ‖ model) over active slots + 0.5 · RPS (ranked probability score) for `score` |
646
  | augmentation | 30 % random permutation of `choice` options (targets permuted consistently) |
647
  | batching | 128 rows / step, kind-stratified (≥ 1/6 per primitive), length-bucketed, micro-batches capped at 24 k padded tokens, gradient checkpointing |
648
  | optimiser | AdamW (fused), β=(0.9, 0.98), lr head 2e-4 / LoRA 1e-4, cosine, warmup 3 %, grad-clip 1.0 for 2,500 steps; then continued on the unseen remainder of the epoch (fresh AdamW state, warmup 2 %, lr ×0.9 → cosine) and stopped after 2,250 more steps at lr ≈ ×0.07 — 4,750 steps ≈ 0.93 epoch in total |
 
653
 
654
  ## Limitations
655
 
656
+ * **System 1 mirrors TypeSafe Jev 1.13, including its mistakes.** This is a distillation, not an independent judge:
657
+ where the teacher was wrong or uncalibrated, so is autotrust/JEV-9B. Published evaluations of the teacher show it is
658
+ unreliable for multi-hop reasoning, arithmetic, dates, counting and adversarial inputs, and the student inherits all
659
+ of that. Confirmed on fresh inputs: the poker shove (0.70 vs the teacher's 0.62).
660
+ * **At 9 B it adds some blind spots of its own.** 11.5 % of its 16-option answers change with option order alone
661
+ (teacher 7.0 %), it keeps 90 % rather than 96 % of the teacher's accuracy at 16 options, and it missed a code-rule
662
+ violation that JEV-27B catches. Prefer [autotrust/JEV-27B](https://huggingface.co/autotrust/JEV-27B) for long option
663
+ lists, unfamiliar task families and code-rule checks.
664
+ * **The two systems are integrated in serving, not in knowledge.** System 1 cannot explain its decisions, and System 2
665
+ is the unmodified base model: it knows nothing about the decisions it is packaged with and was not trained to agree
666
+ with System 1. If you escalate from System 1 to System 2, expect them to disagree sometimes.
667
+ * **"Indistinguishable" is a KL statement on Jev-labelled rows from the 53 training domains.** The corpus has no
668
+ Jev-labelled out-of-distribution set; the OOD figures (KL 0.234, 0.351 for game-state `choice`) are measured against
669
+ programmatic ground truth, and on the independent benchmark the student reaches 90–97 % of Jev's accuracy, not 100 %.
670
+ * **Speed comparisons with the hosted API are not like for like.** Our timings exclude network time; the hosted
671
+ figures are third-party measurements that include it and depend on client concurrency and rate limits.
672
+ * **Choice agreement is capped by teacher ambiguity.** The teacher's `choice` distributions are soft (median top-1
673
+ probability 0.70). On the 14 % of rows where the teacher's top two options are within 0.1 of each other, argmax
674
+ agreement is near chance for *any* faithful mimic (0.46 where the gap is < 0.05). On teacher-decisive rows agreement
675
+ is 0.954, and the student's argmax captures 97.7 % of the teacher probability mass a perfect mimic could (0.693 vs
676
+ 0.709).
677
+ * **Fixed option sets.** `noul` and `score` accept only their canonical options; `choice` accepts 2–16 options. Inputs
678
+ longer than 1,024 tokens are truncated (state only, head 60 % / tail 40 %) at serving unless you raise the limit.
679
+ * **English-centric.** The corpus is English; multilingual behaviour is inherited from the backbone and was not
680
+ systematically measured (the Chinese V2EX examples above are illustrations only).
681
+ * **Placeholder labels in the corpus.** The `yuri_v1` memory-relevance stream is 100 % exact-uniform `[0.5, 0.5]` —
682
+ those rows teach nothing about relevance. The model outputs ≈ 0.5 on them by design; do not use it for
683
+ memory-relevance scoring without further training.
684
+ * **Not for high-stakes decisions.** Use confidence gating: act automatically only above a threshold you validated on
685
+ your own data, and route the rest to System 2, a stronger model, or a human.
686
 
687
  ## Files
688
 
 
690
  model-0000{1..5}-of-00005.safetensors Qwen3.5-9B text backbone incl. lm_head — bit-identical to the base model
691
  (bf16; GDN A_log / gated-norm weights fp32 as in the original), 17.9 GB
692
  model.safetensors.index.json · config.json
693
+ adapter/ System 1 LoRA (peft format, r=16, 40.1 M params, 154 MB) — apply only for decisions
694
  head.safetensors 24-slot decision head (fp32): proj.weight [24, 4096], proj.bias [24]
695
  judge_config.json slot layout, verbalizer token ids, template version, weights_mode=unmerged, provenance
696
  calibration.json per-kind temperatures (+ fit diagnostics)
697
+ adapter_vllm/ the same adapter for vLLM: backbone LoRA (zero-padded to r=32) + decision head as an
698
+ lm_head LoRA, plus decision_head.json (head bias, verbalizer token ids)
699
  tokenizer.json · tokenizer_config.json · chat_template.jinja
700
  reports/ evaluation reports: test-set evaluation, bundle checks, HumanEval per-problem
701
  results, vLLM measurements, real-world tests, training-milestone reviews
 
704
  ## License and acknowledgements
705
 
706
  Weights: **Apache-2.0** (base model `Qwen/Qwen3.5-9B` is Apache-2.0; training corpus
707
+ `SargeDev/jev-distill-corpus-v3` is Apache-2.0, its `openjev_v2` stream additionally CC0). The System One framing and
708
+ the `noul` / `choice` / `score` primitives originate with TypeSafe AI's Jev; autotrust/JEV-9B is an independent
709
+ student model trained on public data and shares no weights, code or affiliation with TypeSafe AI.
710
 
711
  ```bibtex
712
+ @misc{autotrust_jev9b_2026,
713
+ title = {autotrust/JEV-9B: the first integrated System 1 + System 2 open model built with the Blocks of Experts recipe (Qwen3.5-9B; System 1 distilled from TypeSafe Jev 1.13)},
714
+ author = {{AutoTrust AI}},
715
  year = {2026},
716
  url = {https://huggingface.co/autotrust/JEV-9B}
717
  }