Text Classification
Transformers
Safetensors
English
qwen3_5_text
text-generation
system-one
system-two
blocks-of-experts
typed-decisions
decision-model
calibrated-probabilities
knowledge-distillation
jev
noul
choice
score
lora
qwen3_5
dual-head
vllm
Eval Results (legacy)
Instructions to use autotrust/JEV-9B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use autotrust/JEV-9B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="autotrust/JEV-9B")# Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("autotrust/JEV-9B") model = AutoModelForCausalLM.from_pretrained("autotrust/JEV-9B", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Model card: aligned with JEV-27B — first integrated System 1 + System 2 model, Blocks of Experts recipe, KL fidelity, speed vs hosted API, comparison with other reproductions
Browse files
README.md
CHANGED
|
@@ -10,6 +10,8 @@ library_name: transformers
|
|
| 10 |
pipeline_tag: text-classification
|
| 11 |
tags:
|
| 12 |
- system-one
|
|
|
|
|
|
|
| 13 |
- typed-decisions
|
| 14 |
- decision-model
|
| 15 |
- calibrated-probabilities
|
|
@@ -40,13 +42,13 @@ model-index:
|
|
| 40 |
split: test_set_30k
|
| 41 |
metrics:
|
| 42 |
- type: kl_divergence
|
| 43 |
-
name: mean KL(
|
| 44 |
value: 0.0210
|
| 45 |
- type: auroc
|
| 46 |
name: noul AUROC
|
| 47 |
-
value: 0.
|
| 48 |
- type: brier
|
| 49 |
-
name: noul Brier (vs.
|
| 50 |
value: 0.0015
|
| 51 |
- type: mae
|
| 52 |
name: score expected-value MAE (0–5 scale)
|
|
@@ -58,68 +60,265 @@ model-index:
|
|
| 58 |
name: choice top-1 agreement (all rows)
|
| 59 |
value: 0.898
|
| 60 |
- type: accuracy
|
| 61 |
-
name: choice top-1 agreement (
|
| 62 |
value: 0.954
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 63 |
---
|
| 64 |
|
| 65 |
-
# autotrust/JEV-9B
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 66 |
|
| 67 |
-
|
| 68 |
-
> **TypeSafe Jev 1.13** is the hosted, closed model made by TypeSafe AI; it is the *teacher* whose public
|
| 69 |
-
> outputs this model was trained on. **autotrust/JEV-9B** (this repository) is an independent open-weights
|
| 70 |
-
> *student* built by autotrust from the Apache-2.0 corpus
|
| 71 |
-
> [`SargeDev/jev-distill-corpus-v3`](https://huggingface.co/datasets/SargeDev/jev-distill-corpus-v3).
|
| 72 |
-
> autotrust/JEV-9B is not affiliated with, endorsed by, or a product of TypeSafe AI. Throughout this card the
|
| 73 |
-
> teacher is always written *TypeSafe Jev 1.13* and this model is always written *autotrust/JEV-9B*.
|
| 74 |
|
| 75 |
-
|
| 76 |
-
|
| 77 |
-
|
| 78 |
-
|
| 79 |
-
backbone**: the 24-slot decision head (typed decisions) and the original `lm_head` (ordinary text
|
| 80 |
-
generation). The backbone weights on disk are **bit-identical to Qwen3.5-9B**; the decision adapter is a
|
| 81 |
-
separate 154 MB LoRA that is only applied for decisions — so generation quality is exactly the base model's.
|
| 82 |
|
| 83 |
-
|
| 84 |
-
|
| 85 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 86 |
|
| 87 |
-
##
|
| 88 |
|
| 89 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 90 |
|
| 91 |
-
|
|
| 92 |
|---|---|---|
|
| 93 |
-
| **
|
| 94 |
-
| **
|
| 95 |
-
| **
|
| 96 |
-
|
| 97 |
-
|
| 98 |
-
|
| 99 |
-
|
| 100 |
-
|
| 101 |
-
|
| 102 |
-
|
| 103 |
|
| 104 |
-
|
| 105 |
-
|
| 106 |
-
[Evaluation details](#evaluation-details) and in `reports/eval_s2_9b_epoch1.md`.
|
| 107 |
|
| 108 |
-
|
| 109 |
|
| 110 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 111 |
|
| 112 |
-
|
| 113 |
-
|
| 114 |
-
The
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 115 |
|
| 116 |
### Independent benchmark: side by side with TypeSafe Jev 1.13
|
| 117 |
|
| 118 |
[`gazelle93/decision-models-under-pressure`](https://github.com/gazelle93/decision-models-under-pressure) (published
|
| 119 |
25 Sep 2026) asks decision models to pick the right label for real texts from CLINC-150, MTOP, GoEmotions, DBpedia and
|
| 120 |
-
financial tweets
|
| 121 |
-
are human gold labels, none of this data is in our training set, and TypeSafe Jev 1.13's results are published with
|
| 122 |
-
|
| 123 |
|
| 124 |
| | TypeSafe Jev 1.13 (published) | **autotrust/JEV-9B** | autotrust/JEV-27B |
|
| 125 |
|---|---|---|---|
|
|
@@ -129,16 +328,15 @@ it. We re-ran the same items, up to our 16-option limit.
|
|
| 129 |
| Answers changed by shuffling the options alone (16 options, 5 orderings) | 7.0 % | **11.5 %** | 7.4 % |
|
| 130 |
| Time for 14,400 decisions on one B200 | — | **42 s** | 110 s |
|
| 131 |
|
| 132 |
-
|
| 133 |
-
|
| 134 |
-
|
| 135 |
-
the
|
| 136 |
-
|
| 137 |
-
seeded differently, so compare the aggregates, not individual items.
|
| 138 |
|
| 139 |
### Fresh examples (Hacker News and V2EX, 23–25 September 2026)
|
| 140 |
|
| 141 |
-
|
| 142 |
benchmark.
|
| 143 |
|
| 144 |
| task | autotrust/JEV-9B | autotrust/JEV-27B |
|
|
@@ -152,52 +350,51 @@ benchmark.
|
|
| 152 |
|---|---|---|
|
| 153 |
| HN: "Two-tier encryption in the UK" | topic (10 options) | security & privacy · 0.87 |
|
| 154 |
| HN: "Using LLMs to trace alchemical knowledge and decode 17th century letters" | about AI? | P(true) = 0.88 |
|
| 155 |
-
| HN comment: "Why do you talk like that? Are you one of the redditors that has been invading…" | insults or attacks someone? | P(true) = 0.67 |
|
| 156 |
| HN comment: "Please stop this. We've asked you before to observe the guidelines…" | what is it mainly doing? | moderating the discussion · 0.78 |
|
| 157 |
| V2EX: "一个不需要 gemini pro 的完全免费��注册 Muse 的方法 … 邀请码:…" | contains a referral / invite code? | P(true) = 1.00 |
|
| 158 |
| V2EX: "今天中秋节,还要加班的有吗?来报道下" | promotes a product or paid offer? | P(true) = 0.01 |
|
| 159 |
| Diff replacing a parameterised query with `"… WHERE id = " + request.args["id"]` | introduces a security vulnerability? | P(true) = 0.93 (0.17 for a variable rename) |
|
| 160 |
| "I'm not happy with the fit. What are my options here?" | asking for a refund? | P(true) = 0.17 (TypeSafe's docs report 0.22 for Jev on this exact text) |
|
| 161 |
|
| 162 |
-
Where it failed or wavered
|
| 163 |
|
| 164 |
-
* **Code-rule check**:
|
| 165 |
-
|
| 166 |
-
* **Fraud screening**: a CEO wire-transfer (business-email-compromise) message is flagged, but only at 0.56 (
|
| 167 |
* **Comment intent**: "Because it's not a real argument. It's a deflection people use." read as attacking another
|
| 168 |
commenter (0.66) rather than arguing a point.
|
| 169 |
-
* **
|
| 170 |
-
|
| 171 |
-
|
| 172 |
-
|
| 173 |
-
|
| 174 |
-
|
| 175 |
|
| 176 |
Per-example outputs and the benchmark aggregates are in `reports/realworld_9b.json` (the HN and V2EX inputs came from
|
| 177 |
their public APIs on 25 September 2026).
|
| 178 |
|
| 179 |
-
##
|
| 180 |
|
| 181 |
-
**One vLLM engine serves both
|
| 182 |
-
`lm_head` (
|
| 183 |
-
|
| 184 |
-
|
| 185 |
-
|
| 186 |
|
| 187 |
### 1 — Start the server (OpenAI-compatible)
|
| 188 |
|
| 189 |
```bash
|
| 190 |
-
hf download autotrust/JEV-9B --local-dir JEV-9B
|
| 191 |
vllm serve JEV-9B --served-model-name autotrust/JEV-9B \
|
| 192 |
--enable-lora --max-lora-rank 32 --lora-modules jev-decision=JEV-9B/adapter_vllm \
|
| 193 |
--logprobs-mode processed_logprobs --max-model-len 4096
|
| 194 |
```
|
| 195 |
|
| 196 |
-
`--logprobs-mode processed_logprobs` is required: it makes the returned log-probabilities respect
|
| 197 |
-
`
|
| 198 |
-
|
| 199 |
|
| 200 |
-
### 2 —
|
| 201 |
|
| 202 |
```bash
|
| 203 |
curl -s localhost:8000/v1/chat/completions -H 'content-type: application/json' -d '{
|
|
@@ -206,7 +403,11 @@ curl -s localhost:8000/v1/chat/completions -H 'content-type: application/json' -
|
|
| 206 |
"max_tokens": 60, "chat_template_kwargs": {"enable_thinking": false}}'
|
| 207 |
```
|
| 208 |
|
| 209 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 210 |
|
| 211 |
```python
|
| 212 |
import json, math, requests
|
|
@@ -238,10 +439,31 @@ print(decide("noul", "Customer says the parcel arrived damaged and wants their m
|
|
| 238 |
"Is the customer asking for a refund?"))
|
| 239 |
```
|
| 240 |
|
| 241 |
-
Values can differ in the third decimal between runs: vLLM computes in bf16 and results depend slightly on
|
| 242 |
-
|
| 243 |
that vLLM applies cancels out, so the result is exactly the decision head's calibrated distribution.
|
| 244 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 245 |
### Offline / batch (Python API)
|
| 246 |
|
| 247 |
```python
|
|
@@ -251,39 +473,38 @@ from vllm.lora.request import LoRARequest
|
|
| 251 |
llm = LLM("JEV-9B", enable_lora=True, max_lora_rank=32, logprobs_mode="processed_logprobs", max_model_len=4096)
|
| 252 |
decision = LoRARequest("jev-decision", 1, "JEV-9B/adapter_vllm")
|
| 253 |
|
| 254 |
-
gen = llm.generate(["..."], SamplingParams(temperature=0.0, max_tokens=256)) #
|
| 255 |
-
dec = llm.generate([prompt], [SamplingParams(max_tokens=1, temperature=1.0, #
|
| 256 |
allowed_token_ids=ids, logprobs=len(ids))],
|
| 257 |
lora_request=decision) # then + bias, / T as above
|
| 258 |
```
|
| 259 |
|
| 260 |
-
Mixed batches work too: pass a per-request `lora_request` list (`None` for
|
| 261 |
-
|
| 262 |
|
| 263 |
### Measured on one B200
|
| 264 |
|
| 265 |
| workload | PyTorch path | **vLLM** |
|
| 266 |
|---|---|---|
|
| 267 |
-
|
|
| 268 |
-
|
|
| 269 |
-
|
|
| 270 |
-
|
|
| 271 |
| Many questions about one state, `--enable-prefix-caching` | — | +14–20 % throughput |
|
| 272 |
|
| 273 |
Notes:
|
| 274 |
-
* The big win is on
|
| 275 |
-
throughput is about the same as the PyTorch path
|
| 276 |
-
under concurrency, an OpenAI-compatible API, and one engine for both
|
| 277 |
-
* Prefix caching: this architecture mixes Gated DeltaNet and attention layers, and vLLM caches it in blocks of
|
| 278 |
-
|
| 279 |
-
|
| 280 |
-
|
| 281 |
-
identical outputs.
|
| 282 |
* Requires a vLLM build with Qwen3.5 (`qwen3_5`) support, LoRA on `lm_head`, `--logprobs-mode` and
|
| 283 |
`allowed_token_ids`; tested with a vLLM development build from September 2026. Start-up takes 3–8 minutes
|
| 284 |
(CUDA-graph capture with LoRA enabled).
|
| 285 |
|
| 286 |
-
## What
|
| 287 |
|
| 288 |
| kind | question | returns |
|
| 289 |
|---|---|---|
|
|
@@ -304,39 +525,13 @@ D) maintain
|
|
| 304 |
```
|
| 305 |
|
| 306 |
The template is tokenised as one string; the last token's final-norm hidden state goes through a
|
| 307 |
-
**linear fp32 head `H → 24 slots`** (`noul` → slots 0–1, `score` → 2–7, `choice` → 8–23). Inactive slots
|
| 308 |
-
|
| 309 |
-
|
| 310 |
-
|
| 311 |
-
## One backbone, two heads
|
| 312 |
-
|
| 313 |
-
```
|
| 314 |
-
┌── lm_head (248,320 × 4096) ─────────────► next-token logits → text generation
|
| 315 |
-
Qwen3.5-9B text tower ──┤ (pristine base weights; adapter OFF)
|
| 316 |
-
(bit-identical to base)│
|
| 317 |
-
└── + LoRA adapter (154 MB) ── decision head (24 × 4096) ─► calibrated typed decision
|
| 318 |
-
(adapter ON; head rows were initialised from lm_head rows)
|
| 319 |
-
```
|
| 320 |
-
|
| 321 |
-
| mode | weights used | quality |
|
| 322 |
-
|---|---|---|
|
| 323 |
-
| **Text generation** | backbone + `lm_head`, adapter off | exactly Qwen3.5-9B: HumanEval greedy pass@1 70.7 % (116/164), completions byte-identical to the base model |
|
| 324 |
-
| **Typed decisions** | backbone + `adapter/` + `head.safetensors` | the numbers in [Headline results](#headline-results) |
|
| 325 |
-
|
| 326 |
-
Why the adapter is shipped unmerged: folding the LoRA into the backbone would let one weight set serve both
|
| 327 |
-
heads, but it costs generation quality — the merged backbone with the original `lm_head` scores **61.6 %**
|
| 328 |
-
(101/164) on HumanEval, a 9-point drop, even though its perplexity on prose barely moves (3.15 → 3.30).
|
| 329 |
-
Keeping the base pristine and applying the 40 M-parameter adapter only on the decision path removes that
|
| 330 |
-
trade-off. For decision serving the adapter is merged *in memory* at start-up, so decision latency is the
|
| 331 |
-
same as a merged bundle.
|
| 332 |
|
| 333 |
-
##
|
| 334 |
|
| 335 |
-
###
|
| 336 |
-
|
| 337 |
-
See [Inference with vLLM](#inference-with-vllm-recommended) above.
|
| 338 |
-
|
| 339 |
-
### Option B — plain `transformers` + `peft`
|
| 340 |
|
| 341 |
```python
|
| 342 |
import json, torch
|
|
@@ -349,13 +544,13 @@ repo = "autotrust/JEV-9B"
|
|
| 349 |
tok = AutoTokenizer.from_pretrained(repo)
|
| 350 |
base = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="cuda") # == Qwen3.5-9B text model
|
| 351 |
|
| 352 |
-
# ---
|
| 353 |
msgs = [{"role": "user", "content": "In two sentences, what is safety stock?"}]
|
| 354 |
enc = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True).to("cuda")
|
| 355 |
out = base.generate(**enc, max_new_tokens=80)
|
| 356 |
print(tok.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True))
|
| 357 |
|
| 358 |
-
# ---
|
| 359 |
model = PeftModel.from_pretrained(base, repo, subfolder="adapter").merge_and_unload()
|
| 360 |
head = load_file(hf_hub_download(repo, "head.safetensors"))
|
| 361 |
cfg = json.load(open(hf_hub_download(repo, "judge_config.json")))
|
|
@@ -379,69 +574,31 @@ print(decide("noul", "Customer says the parcel arrived damaged and wants their m
|
|
| 379 |
# {'false': 0.009, 'true': 0.991}
|
| 380 |
```
|
| 381 |
|
| 382 |
-
`options` are validated: `noul` must be `["false","true"]`, `score` must be `["0".."5"]`, `choice` takes
|
| 383 |
-
|
| 384 |
-
inside `with model.disable_adapter():`.
|
| 385 |
-
|
| 386 |
-
## Why start from a pretrained LLM
|
| 387 |
-
|
| 388 |
-
A System One judge is "just a classifier", so one could train a small encoder from scratch or fine-tune a
|
| 389 |
-
BERT-class model. autotrust/JEV-9B starts from a pretrained 9 B LLM instead, and that choice — not the 3 GPU-hours of
|
| 390 |
-
distillation — is where most of the result above comes from:
|
| 391 |
-
|
| 392 |
-
1. **Pretraining does most of the work; distillation only sharpens.** The 24-slot head is *initialised from
|
| 393 |
-
the backbone's own `lm_head` rows* for the verbalizer tokens (`false/true`, `0`–`5`, `A`–`P`), so at
|
| 394 |
-
step 0 the head's output is *exactly* the pretrained model's zero-shot restricted next-token distribution
|
| 395 |
-
(verified to |Δp| < 1e-5; measured 8.6e-07). Without seeing a single label that starting point already
|
| 396 |
-
agrees with TypeSafe Jev 1.13 on 53 % of choice questions and reaches noul AUROC 0.82; distillation takes it to
|
| 397 |
-
90 % / 0.996. World knowledge, instruction reading and long-context handling are inherited, not learned.
|
| 398 |
-
|
| 399 |
-
2. **Extreme sample- and compute-efficiency.** Only **40.2 M parameters** are trained — LoRA r=16 on the
|
| 400 |
-
decoder projections plus the 98 k-parameter head, 0.5 % of the 8.0 B text backbone. KL 0.028 was reached after
|
| 401 |
-
**0.49 epoch (≈ 320 k rows, ≈ 1.7 GPU-hours on one B200)** and KL 0.021 after 0.93 epoch (≈ 3 GPU-hours);
|
| 402 |
-
after the first 64 k rows validation KL was already below 0.10.
|
| 403 |
-
|
| 404 |
-
3. **Transfer to unseen task families.** On the Open-Jev *OOD* split (game-state and workflow-policy
|
| 405 |
-
decisions never seen in training) autotrust/JEV-9B reaches KL 0.234 and top-1 0.918 — a pretrained LLM reads the
|
| 406 |
-
*content* of a new task instead of matching surface patterns of the 53 training domains.
|
| 407 |
-
|
| 408 |
-
4. **Reads real, long, structured states.** States in the corpus are prose, JSON game states, policy
|
| 409 |
-
documents and retrieved passages up to 856 tokens; the backbone supports far longer. BERT-class encoders
|
| 410 |
-
stop at 512 tokens and lack the code/JSON pretraining to parse them.
|
| 411 |
-
|
| 412 |
-
5. **Reads options, not positions.** With 30 % option-permutation augmentation during distillation, the
|
| 413 |
-
top-1 flip rate under shuffled `choice` options is 3.9 %; the same backbone before distillation flips
|
| 414 |
-
38 % of the time. The pretrained attention binds the letter to the option text almost immediately.
|
| 415 |
-
|
| 416 |
-
6. **Calibration falls out for free.** Distilling *full* teacher distributions with KL (plus an ordinal RPS
|
| 417 |
-
term for `score`) on a pretrained backbone gives fitted temperatures of **1.002 / 0.984 / 1.012**: the
|
| 418 |
-
model is calibrated to the teacher with no post-hoc correction (ECE 0.0007).
|
| 419 |
-
|
| 420 |
-
7. **The recipe scales without code changes.** The head-initialisation identity holds for any `qwen3_5`
|
| 421 |
-
checkpoint without logit soft-capping; the equivalence gate also passes on Qwen3.8-27B (4.5e-07). 9 B was
|
| 422 |
-
kept because it already meets the distributional targets.
|
| 423 |
|
| 424 |
## Evaluation details
|
| 425 |
|
| 426 |
-
### Additional metrics (`test_set_30k`, temperature applied)
|
| 427 |
|
| 428 |
| metric | autotrust/JEV-9B |
|
| 429 |
|---|---|
|
| 430 |
-
| `noul` Brier score against the
|
| 431 |
| `score` ranked probability score (lower is better) | 0.0085 |
|
| 432 |
| Fitted temperatures noul / choice / score | 1.002 / 0.984 / 1.012 |
|
| 433 |
| Top-1 flip rate when `choice` options are shuffled (1,000 rows × 4 permutations) | 3.9 % |
|
| 434 |
| Out-of-distribution split — top-1 agreement · `noul` AUROC | 0.918 · 0.989 |
|
| 435 |
| Throughput — batch of 128 requests on one B200 | 2.5 ms per decision (≈ 400 decisions/s) |
|
| 436 |
-
|
|
| 437 |
|
| 438 |
-
###
|
| 439 |
|
| 440 |
| weights | pass@1 | note |
|
| 441 |
|---|---|---|
|
| 442 |
| Qwen3.5-9B (base) | 70.7 % (116/164) | same loader and protocol as below |
|
| 443 |
-
| **autotrust/JEV-9B —
|
| 444 |
-
| LoRA folded into the backbone + base `lm_head` (*not shipped*) | 61.6 % (101/164) | why the
|
| 445 |
|
| 446 |
### Per source × primitive (`test_set_30k`, temperature applied)
|
| 447 |
|
|
@@ -450,17 +607,17 @@ distillation — is where most of the result above comes from:
|
|
| 450 |
| yuri_v3 — synthetic operational scenarios, labelled by TypeSafe Jev 1.13 | noul | 8,537 | 0.005 | 0.961 | 0.001 | 0.994 | — |
|
| 451 |
| yuri_v3 | choice | 8,312 | 0.028 | 0.902 | 0.002 | — | — |
|
| 452 |
| yuri_v3 | score | 8,527 | 0.023 | 0.883 | 0.002 | — | 0.103 |
|
| 453 |
-
| openjev_v2 — Open-Jev programmatic tasks | noul | 1,432 | 0.004 | 0.998 | 0.003 | 1.000 | — |
|
| 454 |
| openjev_v2 | choice | 887 | 0.176 | 0.857 | 0.020 | — | — |
|
| 455 |
| yuri_v1 — placeholder `[0.5, 0.5]` labels (see Limitations) | noul | 2,260 | 0.000 | — | 0.005 | — | — |
|
| 456 |
|
| 457 |
-
OOD split (13,058 Open-Jev rows from task families not in training): KL 0.234, top-1 0.918, noul
|
| 458 |
-
0.989; `choice` KL 0.351 / top-1 0.837 (game-state decisions are the hardest slice).
|
| 459 |
|
| 460 |
-
Choice option-permutation consistency (1,000 rows × 4 random permutations): mean max |Δp| 0.024, p90 0.055,
|
| 461 |
-
|
| 462 |
|
| 463 |
-
###
|
| 464 |
|
| 465 |
Validation KL on a fixed 4 k-row subset; `test_set_30k` metrics after calibration.
|
| 466 |
|
|
@@ -473,18 +630,19 @@ Validation KL on a fixed 4 k-row subset; `test_set_30k` metrics after calibratio
|
|
| 473 |
| step 500 | 64 k | 0.094 | 0.081 | 0.817 | 0.224 | 0.977 | 0.022 |
|
| 474 |
| untrained backbone with the initialised head (reference point, not the model) | 0 | 0.485 | 0.510 | 0.532 | 1.130 | 0.824 | 0.094 |
|
| 475 |
|
| 476 |
-
Annealing matters: the v0.7.0 cool-down (500 steps, lr ×0.9 → ×0.02 from step 2000) lowered KL by 25 %;
|
| 477 |
-
|
| 478 |
-
|
| 479 |
|
| 480 |
## Training details
|
| 481 |
|
| 482 |
| item | value |
|
| 483 |
|---|---|
|
| 484 |
-
| teacher / data | TypeSafe Jev 1.13 full output distributions (
|
| 485 |
| backbone | `Qwen/Qwen3.5-9B` text tower only (vision tower and MTP head dropped), bf16, frozen |
|
| 486 |
-
| trainable | LoRA r=16, α=32, dropout 0.05 on `in_proj_qkv, in_proj_z, out_proj, q/k/v/o_proj, gate/up/down_proj` (40.1 M, shipped unmerged in `adapter/`) + 24-slot head (98 k, fp32, initialised from `lm_head` rows) |
|
| 487 |
-
|
|
|
|
|
| 488 |
| augmentation | 30 % random permutation of `choice` options (targets permuted consistently) |
|
| 489 |
| batching | 128 rows / step, kind-stratified (≥ 1/6 per primitive), length-bucketed, micro-batches capped at 24 k padded tokens, gradient checkpointing |
|
| 490 |
| optimiser | AdamW (fused), β=(0.9, 0.98), lr head 2e-4 / LoRA 1e-4, cosine, warmup 3 %, grad-clip 1.0 for 2,500 steps; then continued on the unseen remainder of the epoch (fresh AdamW state, warmup 2 %, lr ×0.9 → cosine) and stopped after 2,250 more steps at lr ≈ ×0.07 — 4,750 steps ≈ 0.93 epoch in total |
|
|
@@ -495,30 +653,36 @@ another 24 % and added 1.4 points of choice agreement. Checkpoints taken at high
|
|
| 495 |
|
| 496 |
## Limitations
|
| 497 |
|
| 498 |
-
* **
|
| 499 |
-
the teacher was wrong or uncalibrated, so is autotrust/JEV-9B. Published evaluations of the teacher show it is
|
| 500 |
-
for multi-hop reasoning, arithmetic, dates, counting
|
| 501 |
-
|
| 502 |
-
|
| 503 |
-
|
| 504 |
-
|
| 505 |
-
|
| 506 |
-
|
| 507 |
-
|
| 508 |
-
|
| 509 |
-
|
| 510 |
-
|
| 511 |
-
|
| 512 |
-
* **
|
| 513 |
-
|
| 514 |
-
|
| 515 |
-
|
| 516 |
-
|
| 517 |
-
|
| 518 |
-
|
| 519 |
-
|
| 520 |
-
|
| 521 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 522 |
|
| 523 |
## Files
|
| 524 |
|
|
@@ -526,12 +690,12 @@ another 24 % and added 1.4 points of choice agreement. Checkpoints taken at high
|
|
| 526 |
model-0000{1..5}-of-00005.safetensors Qwen3.5-9B text backbone incl. lm_head — bit-identical to the base model
|
| 527 |
(bf16; GDN A_log / gated-norm weights fp32 as in the original), 17.9 GB
|
| 528 |
model.safetensors.index.json · config.json
|
| 529 |
-
adapter/
|
| 530 |
head.safetensors 24-slot decision head (fp32): proj.weight [24, 4096], proj.bias [24]
|
| 531 |
judge_config.json slot layout, verbalizer token ids, template version, weights_mode=unmerged, provenance
|
| 532 |
calibration.json per-kind temperatures (+ fit diagnostics)
|
| 533 |
-
adapter_vllm/ the same adapter for vLLM: backbone LoRA (zero-padded to r=32) + decision head as an
|
| 534 |
-
plus decision_head.json (head bias, verbalizer token ids)
|
| 535 |
tokenizer.json · tokenizer_config.json · chat_template.jinja
|
| 536 |
reports/ evaluation reports: test-set evaluation, bundle checks, HumanEval per-problem
|
| 537 |
results, vLLM measurements, real-world tests, training-milestone reviews
|
|
@@ -540,14 +704,14 @@ reports/ evaluation reports: test-set evaluation,
|
|
| 540 |
## License and acknowledgements
|
| 541 |
|
| 542 |
Weights: **Apache-2.0** (base model `Qwen/Qwen3.5-9B` is Apache-2.0; training corpus
|
| 543 |
-
`SargeDev/jev-distill-corpus-v3` is Apache-2.0, its `openjev_v2` stream additionally CC0). The System One
|
| 544 |
-
|
| 545 |
-
|
| 546 |
|
| 547 |
```bibtex
|
| 548 |
-
@misc{
|
| 549 |
-
title = {autotrust/JEV-9B:
|
| 550 |
-
author = {
|
| 551 |
year = {2026},
|
| 552 |
url = {https://huggingface.co/autotrust/JEV-9B}
|
| 553 |
}
|
|
|
|
| 10 |
pipeline_tag: text-classification
|
| 11 |
tags:
|
| 12 |
- system-one
|
| 13 |
+
- system-two
|
| 14 |
+
- blocks-of-experts
|
| 15 |
- typed-decisions
|
| 16 |
- decision-model
|
| 17 |
- calibrated-probabilities
|
|
|
|
| 42 |
split: test_set_30k
|
| 43 |
metrics:
|
| 44 |
- type: kl_divergence
|
| 45 |
+
name: mean KL(target ‖ model), all test rows (25,376 of 29,955 targets are TypeSafe Jev 1.13 distributions)
|
| 46 |
value: 0.0210
|
| 47 |
- type: auroc
|
| 48 |
name: noul AUROC
|
| 49 |
+
value: 0.996
|
| 50 |
- type: brier
|
| 51 |
+
name: noul Brier (vs. target probability, all rows)
|
| 52 |
value: 0.0015
|
| 53 |
- type: mae
|
| 54 |
name: score expected-value MAE (0–5 scale)
|
|
|
|
| 60 |
name: choice top-1 agreement (all rows)
|
| 61 |
value: 0.898
|
| 62 |
- type: accuracy
|
| 63 |
+
name: choice top-1 agreement (decisive-target rows, top-2 gap ≥ 0.1)
|
| 64 |
value: 0.954
|
| 65 |
+
- task:
|
| 66 |
+
type: text-generation
|
| 67 |
+
name: code generation — System 2 path (base lm_head, adapter off)
|
| 68 |
+
dataset:
|
| 69 |
+
type: openai/openai_humaneval
|
| 70 |
+
name: HumanEval
|
| 71 |
+
split: test
|
| 72 |
+
metrics:
|
| 73 |
+
- type: pass@1
|
| 74 |
+
name: pass@1 (greedy, completion-style prompt)
|
| 75 |
+
value: 0.707
|
| 76 |
---
|
| 77 |
|
| 78 |
+
# autotrust/JEV-9B
|
| 79 |
+
|
| 80 |
+
### AutoTrust's first integrated System 1 + System 2 open model, built with the Blocks of Experts recipe
|
| 81 |
+
|
| 82 |
+
**Fast, calibrated System 1 decisions that are indistinguishable from the closed TypeSafe Jev 1.13 by KL, and
|
| 83 |
+
deliberate System 2 generation and reasoning from an untouched Qwen3.5-9B — one set of weights, one vLLM engine,
|
| 84 |
+
routed per request. The fastest model of the family: it answers a single decision in about a third of the time the
|
| 85 |
+
hosted API takes. Its successor, [autotrust/JEV-27B](https://huggingface.co/autotrust/JEV-27B), is closer to Jev,
|
| 86 |
+
transfers better to unseen tasks and has a stronger System 2.**
|
| 87 |
+
|
| 88 |
+
## At a glance
|
| 89 |
+
|
| 90 |
+
**Integrated System 1 + System 2, first generation.** JEV-9B is AutoTrust's first open model to serve both modes of
|
| 91 |
+
thinking from a single set of weights; the second generation is
|
| 92 |
+
[autotrust/JEV-27B](https://huggingface.co/autotrust/JEV-27B). *System 1* answers typed questions (`noul` yes/no ·
|
| 93 |
+
`choice` over 2–16 options · `score` on a 0–5 scale) in one forward pass and returns a calibrated probability
|
| 94 |
+
distribution. *System 2* is ordinary text generation with step-by-step reasoning (thinking mode). Both run on the
|
| 95 |
+
same backbone in the same engine, and a request chooses its system.
|
| 96 |
+
|
| 97 |
+
**Blocks of Experts recipe.** Rather than fine-tuning one monolithic model, the Blocks of Experts (BoE) recipe keeps a
|
| 98 |
+
strong pretrained model as a frozen expert block and adds a small, detachable expert block trained for one capability.
|
| 99 |
+
In JEV-9B the System 2 block is Qwen3.5-9B, bit-identical to the release; the System 1 block is 40.2 M trained
|
| 100 |
+
parameters (0.5 % of the backbone), trained in ≈ 3 hours on one B200. Because the blocks stay separate, adding
|
| 101 |
+
System 1 costs System 2 nothing: HumanEval is 70.7 % before and after, with all 164 completions byte-identical. Folding
|
| 102 |
+
the same block into the backbone instead would have cost 9 points (61.6 %).
|
| 103 |
+
|
| 104 |
+
**Indistinguishable from the closed original on System 1, by KL.** On the 25,376 held-out questions (53 domains)
|
| 105 |
+
whose labels are TypeSafe Jev 1.13's own output distributions, the mean KL divergence is **≈ 0.019** (0 = identical).
|
| 106 |
+
An observer who sees sampled decisions gains on average 0.019 nats of evidence per decision about which model produced
|
| 107 |
+
it, so it takes about 54 decisions to gather a single nat. The fidelity extends to the teacher's mistakes (see
|
| 108 |
+
[System 1: indistinguishable from TypeSafe Jev 1.13](#system-1-indistinguishable-from-typesafe-jev-113-by-kl)).
|
| 109 |
+
Among the open Jev reproductions we could find, only the JEV models publish this distribution-level measure
|
| 110 |
+
(see [How JEV-9B compares with other open Jev reproductions](#how-jev-9b-compares-with-other-open-jev-reproductions)).
|
| 111 |
+
|
| 112 |
+
**Faster than the hosted API.** On one B200, a single decision takes a median ≈ 90 ms, against 238–301 ms measured
|
| 113 |
+
independently for the hosted TypeSafe Jev 1.13 API, and one GPU sustains about 15× the decisions per second an
|
| 114 |
+
independent benchmark achieved against that API (see [Speed](#speed-vs-the-hosted-typesafe-jev-113)).
|
| 115 |
+
|
| 116 |
+
**The fast member of the family; JEV-27B is the closer one.** Same recipe, same API: JEV-9B is 2.6× faster than
|
| 117 |
+
JEV-27B on the same benchmark and its weights are a third of the size (18 GB vs 54 GB). JEV-27B lowers mean KL to Jev's
|
| 118 |
+
distributions from ≈ 0.019 to ≈ 0.017, more than halves KL on unseen task families (0.234 → 0.104), keeps 96 % instead
|
| 119 |
+
of 90 % of the teacher's accuracy on an independent 16-option benchmark, and scores 78.0 % instead of 70.7 % on
|
| 120 |
+
HumanEval (see [JEV-9B vs JEV-27B](#jev-9b-vs-jev-27b)).
|
| 121 |
+
|
| 122 |
+
> **Two models, two organisations.** **TypeSafe Jev 1.13** is the hosted, closed-source model made by TypeSafe AI; it
|
| 123 |
+
> is the *teacher* whose published output distributions this model was trained on. **autotrust/JEV-9B** (this
|
| 124 |
+
> repository) is an independent open-weights *student* built by AutoTrust AI from the Apache-2.0 corpus
|
| 125 |
+
> [`SargeDev/jev-distill-corpus-v3`](https://huggingface.co/datasets/SargeDev/jev-distill-corpus-v3). It is not
|
| 126 |
+
> affiliated with, endorsed by, or a product of TypeSafe AI, and shares no weights or code with it.
|
| 127 |
|
| 128 |
+
## Headline results
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 129 |
|
| 130 |
+
System 1 numbers are on the held-out `test_set_30k` of `jev-distill-corpus-v3`. Its 29,955 rows come from three
|
| 131 |
+
sources: 25,376 rows labelled with TypeSafe Jev 1.13's own output distributions (`yuri_v3`), 2,319 Open-Jev rows with
|
| 132 |
+
programmatic ground-truth labels (`openjev_v2`), and 2,260 placeholder rows (`yuri_v1`). Rows marked *Jev-labelled* use
|
| 133 |
+
only the first group.
|
|
|
|
|
|
|
|
|
|
| 134 |
|
| 135 |
+
| | What is measured | autotrust/JEV-9B | How to read it |
|
| 136 |
+
|---|---|---|---|
|
| 137 |
+
| **System 1** | Mean KL divergence from TypeSafe Jev 1.13's distributions, Jev-labelled rows, 0 = identical | **≈ 0.019** | Indistinguishable from the teacher's decisions at this resolution: ≈ 54 sampled decisions to gather one nat of evidence |
|
| 138 |
+
| | Mean KL to all test targets (Jev, programmatic and placeholder labels) | **0.021** | The figure in the model index above |
|
| 139 |
+
| | Yes/no AUROC (`noul`), Jev-labelled rows | **0.994** | Ranks true vs. false almost perfectly (0.996 over all rows) |
|
| 140 |
+
| | Choice top-1 agreement with Jev, Jev-labelled rows | **90.2 %** | 95.4 % over all rows where the target's top two options differ by ≥ 0.1; on near ties any faithful copy agrees about half the time |
|
| 141 |
+
| | Rating error (`score`, 0–5 scale), mean absolute error of the expected rating | **0.103** | About one tenth of a rating step |
|
| 142 |
+
| | Expected calibration error | **0.0007** | A stated 80 % is an 80 %; fitted temperatures ≈ 1.00, no post-hoc correction needed |
|
| 143 |
+
| | KL to the programmatic labels of task families never seen in training (Open-Jev OOD split) | **0.234** | Transfer to new tasks; these labels are ground truth, not Jev's outputs. JEV-27B: 0.104 |
|
| 144 |
+
| | Independent benchmark with human gold labels, 16 options | **90 % of the teacher** (0.694 vs 0.769) | 94–97 % of the teacher at 2, 4 and 8 options; see [Benchmark highlights](#benchmark-highlights) |
|
| 145 |
+
| **System 2** | HumanEval pass@1, greedy | **70.7 %** | Identical to Qwen3.5-9B (116/164); all 164 completions byte-identical to the base model |
|
| 146 |
+
| **Speed** | Single decision, median, one B200 | **≈ 90 ms** | Hosted TypeSafe Jev 1.13, measured independently: 238 ms mean, 291–301 ms median |
|
| 147 |
+
| | Decisions per second on the independent benchmark, one B200 | **≈ 340** | ≈ 15× the 23 per second measured against the hosted API; see [Speed](#speed-vs-the-hosted-typesafe-jev-113) |
|
| 148 |
+
| | Batched, 128 decisions per batch | **2.5 ms** per decision | With vLLM: 205 decisions/s over HTTP at 256 concurrent clients, text generation ≈ 50× faster than the PyTorch path |
|
| 149 |
+
| **Efficiency** | Trained parameters | **40.2 M** (0.5 % of 7.9 B) | ≈ 3 B200-hours, 0.93 epoch ��� 608 k rows |
|
| 150 |
+
|
| 151 |
+
## JEV-9B vs JEV-27B
|
| 152 |
+
|
| 153 |
+
JEV-9B was AutoTrust's first integrated System 1 + System 2 model.
|
| 154 |
+
[autotrust/JEV-27B](https://huggingface.co/autotrust/JEV-27B), the second generation, uses the same recipe, code,
|
| 155 |
+
hyper-parameters, API and two-block packaging; only the backbone and memory settings changed. Both are evaluated on the
|
| 156 |
+
same held-out test set and the same independent benchmark.
|
| 157 |
+
|
| 158 |
+
<p align="center">
|
| 159 |
+
<a href="https://huggingface.co/autotrust/JEV-27B/blob/main/27b-2.jpg"><img src="https://huggingface.co/autotrust/JEV-27B/resolve/main/27b-2.jpg" alt="JEV family benchmark highlights: KL to TypeSafe Jev 1.13 by question type, accuracy as a percentage of Jev on an independent benchmark, HumanEval for the System 2 path, and speed against the hosted API, for JEV-9B (light bars) and JEV-27B (dark bars)" width="100%"></a>
|
| 160 |
+
<br>
|
| 161 |
+
<sub><b>JEV family benchmark highlights</b> (chart from the JEV-27B repository; light bars = JEV-9B). A · KL to Jev by question type · B · accuracy as % of Jev on an independent benchmark · C · System 2 unchanged · D · speed vs the hosted API · click to enlarge</sub>
|
| 162 |
+
</p>
|
| 163 |
+
|
| 164 |
+
| | **JEV-9B** | JEV-27B | JEV-27B vs JEV-9B |
|
| 165 |
+
|---|---|---|---|
|
| 166 |
+
| Backbone | Qwen3.5-9B | Qwen3.8-27B | |
|
| 167 |
+
| **System 1** — mean KL to TypeSafe Jev 1.13, Jev-labelled rows | ≈ 0.019 | **≈ 0.017** | ≈ −11 % |
|
| 168 |
+
| Mean KL to all test targets | 0.021 | **0.019** | −11 % |
|
| 169 |
+
| KL to ground-truth labels, unseen task families (OOD) | 0.234 | **0.104** | −56 % |
|
| 170 |
+
| Top-1 accuracy, unseen task families (OOD) | 0.918 | **0.942** | +2.4 pts |
|
| 171 |
+
| Choice top-1 agreement with Jev, Jev-labelled rows | 90.2 % | **90.5 %** | +0.3 pts |
|
| 172 |
+
| Rating error (`score` MAE, all Jev-labelled) | 0.103 | **0.098** | −5 % |
|
| 173 |
+
| Top-1 flips under option shuffle (test set) | 3.9 % | **2.9 %** | −1.0 pt |
|
| 174 |
+
| Yes/no AUROC (`noul`), Jev-labelled rows | 0.994 | **0.995** | +0.001 |
|
| 175 |
+
| Calibration error (ECE) | **0.0007** | 0.0009 | JEV-9B slightly lower; both below 0.001 |
|
| 176 |
+
| Independent benchmark, 16 options — % of teacher accuracy | 90 % | **96 %** | +6 pts |
|
| 177 |
+
| Independent benchmark — answers changed by option order alone (teacher: 7.0 %) | 11.5 % | **7.4 %** | JEV-27B is close to the teacher's 7.0 % |
|
| 178 |
+
| **System 2** — HumanEval pass@1 (greedy) | 70.7 % | **78.0 %** | +7.3 pts |
|
| 179 |
+
| Latency on one B200 — single request / batched | **≈ 90 ms / 2.5 ms** | 137 ms / 4.2 ms | JEV-9B is faster |
|
| 180 |
+
| Benchmark throughput — 14,400 decisions on one B200 | **42 s** | 110 s | JEV-9B is 2.6× faster |
|
| 181 |
+
| Download size (backbone + adapter) | **18 GB** | 54 GB | |
|
| 182 |
+
| Trained parameters / compute | 40.2 M / ≈ 3 B200-hours | 108.9 M / ≈ 9.2 B200-hours | |
|
| 183 |
+
|
| 184 |
+
On the fresh Hacker News, V2EX and community examples (illustrations, not a benchmark), JEV-9B got 92 of 96 decisions
|
| 185 |
+
right against 95 of 96 for JEV-27B. The difference is on the harder tasks: JEV-9B misses a TypeScript port that
|
| 186 |
+
breaks a "branded, range-checked integer" rule (0.33; JEV-27B 0.93) and flags a CEO wire-transfer fraud message with
|
| 187 |
+
less confidence (0.56; JEV-27B 0.84).
|
| 188 |
+
|
| 189 |
+
**Which to pick.** For routing, moderation, topic triage and short option lists, JEV-9B gives nearly the same answers
|
| 190 |
+
2.6× faster (14,400 benchmark decisions in 42 s vs 110 s on one B200) with a third of the weight memory. For long option lists
|
| 191 |
+
(more than about 8), unfamiliar task families, code-rule checks, fraud screening, or when the System 2 path matters, use
|
| 192 |
+
[autotrust/JEV-27B](https://huggingface.co/autotrust/JEV-27B).
|
| 193 |
|
| 194 |
+
## How JEV-9B compares with other open Jev reproductions
|
| 195 |
|
| 196 |
+
Dozens of open reproductions of TypeSafe Jev appeared within weeks of its launch; the community
|
| 197 |
+
[Jev Decision Index](https://huggingface.co/spaces/multimodalart/jev-decision-index) (formerly the Jev Reproductions
|
| 198 |
+
Tracker) evaluates 55 of them. Most are trained on human or programmatic gold labels, or on their own synthetic data, so
|
| 199 |
+
they aim to match or beat Jev's accuracy rather than reproduce its probabilities. "Closest to Jev" therefore depends on
|
| 200 |
+
how closeness is measured:
|
| 201 |
|
| 202 |
+
| measure of closeness to TypeSafe Jev 1.13 | published results (snapshot of 25 September 2026) | where JEV-9B stands |
|
| 203 |
|---|---|---|
|
| 204 |
+
| **Distribution level:** KL to Jev's own output distributions on held-out rows | JEV-27B ≈ 0.017 and JEV-9B ≈ 0.019 on 25,376 Jev-labelled rows. We found no other open reproduction that publishes this measure. | Second lowest published, after JEV-27B |
|
| 205 |
+
| **Accuracy relative to Jev** on [`decision-models-under-pressure`](https://github.com/gazelle93/decision-models-under-pressure), 16 options, human gold labels | JEV-27B 96 % · JEV-9B 90 % · Laya 90 % · DeBERTa-v3-large zero-shot 90 % · DeBERTa-v3-base zero-shot 83 % · GLiClass-large 81 % · bge-large 73 % · gte-large 69 % | Level with the best of the other models measured there; JEV-27B is closer (JEV rows are AutoTrust re-runs of the same items; the others were run by the benchmark's author) |
|
| 206 |
+
| **Score parity on community leaderboards** | [Decision Index 0.2](https://huggingface.co/spaces/multimodalart/jev-decision-index): Jev 51.67, AutoJev-27B 50.94. [JevBench v1.4.2](https://github.com/fstandhartinger/jevbench): decider-4b v2 64.13, Jev 63.29, JevK5 62.04. [Open-Jev](https://zefan-cai.github.io/open-jev/benchmarks/) public JevBench subset: Jev 200/231, Open-Jev 27B v1.1 197/231 | Not yet evaluated |
|
| 207 |
+
|
| 208 |
+
On the evidence published today, the two JEV models are the closest open models to TypeSafe Jev 1.13 at the level of
|
| 209 |
+
output distributions, with JEV-9B second to JEV-27B. On the independent benchmark JEV-9B is level with the best of the
|
| 210 |
+
other models measured there, not ahead of them. It has not yet been run on the Decision Index or JevBench, where
|
| 211 |
+
AutoJev-27B scores within about one point of Jev and decider-4b v2 edges ahead of it, so we do not claim it is the
|
| 212 |
+
closest by every measure. Note that some reproductions report beating Jev on their own test sets (AutoJev-27B reports
|
| 213 |
+
84.60 % against Jev's 82.79 %); that is a different goal from reproducing Jev's behaviour.
|
| 214 |
|
| 215 |
+
*Not to be confused with AutoJev-27B (`denis-pplx/autojev-27b`), an unrelated Qwen3.8-27B decision model trained with
|
| 216 |
+
full-weight SFT on its own data.*
|
|
|
|
| 217 |
|
| 218 |
+
## Speed vs the hosted TypeSafe Jev 1.13
|
| 219 |
|
| 220 |
+
TypeSafe does not publish Jev's size or hardware; it reports 70–500 ms end to end. Independent measurements, and ours:
|
| 221 |
+
|
| 222 |
+
| | TypeSafe Jev 1.13, hosted API | **JEV-9B, one B200** | JEV-27B, one B200 |
|
| 223 |
+
|---|---|---|---|
|
| 224 |
+
| One decision, single request | 238 ms mean over 29,600 calls ([`decision-models-under-pressure`](https://github.com/gazelle93/decision-models-under-pressure)); 291–301 ms median on three workloads ([Open-Jev](https://zefan-cai.github.io/open-jev/benchmarks/)) | **≈ 90 ms** median (87 ms) | 137 ms median |
|
| 225 |
+
| Decisions per second on `decision-models-under-pressure` | 23, with 5 client workers and one question per call | **≈ 340** (14,400 in 42 s) | ≈ 130 (14,400 in 110 s) |
|
| 226 |
+
| Batched, 128 decisions per batch | — | **2.5 ms** per decision | 4.2 ms per decision |
|
| 227 |
+
|
| 228 |
+
So JEV-9B answers a single decision in roughly a third of the time (JEV-27B in roughly half), and one GPU sustains
|
| 229 |
+
about 15× (JEV-27B: about 6×) the throughput the benchmark's author achieved against the hosted API. Read these with
|
| 230 |
+
the caveats: our latencies are measured on the serving host with no network hop, while the hosted numbers include
|
| 231 |
+
internet, TLS and queueing; hosted throughput depends on client concurrency and the API's rate limits; Jev's latency is
|
| 232 |
+
roughly flat in the number of questions per request, so bundling questions narrows the throughput gap; and our figures
|
| 233 |
+
are self-reported while Jev's come from third parties. The two throughput runs use the same benchmark but not an
|
| 234 |
+
identical call set (ours stops at 16 options).
|
| 235 |
+
|
| 236 |
+
## System 1: indistinguishable from TypeSafe Jev 1.13, by KL
|
| 237 |
+
|
| 238 |
+
**What the number means.** KL(Jev ‖ model) is the expected log-likelihood ratio, per sampled decision, between
|
| 239 |
+
TypeSafe Jev 1.13 and the student when the decision comes from Jev. On the 25,376 held-out rows whose targets are
|
| 240 |
+
Jev's own output distributions, the mean is ≈ 0.019 nats (computed from the per-slice values below, which are
|
| 241 |
+
published to three decimals): one decision carries almost no evidence about which of the two models produced it, and
|
| 242 |
+
an observer needs about 1 / KL ≈ 54 independent decisions to accumulate one nat (a likelihood ratio of about e ≈ 2.7 : 1).
|
| 243 |
+
|
| 244 |
+
For scale, Jev is not deterministic itself: an independent study found it changes its answer on 4.3 % of repeated,
|
| 245 |
+
identical 64-option calls, and it returns probabilities rounded to two decimals, which is the resolution of the
|
| 246 |
+
targets used here.
|
| 247 |
+
|
| 248 |
+
| Jev-labelled slice (`yuri_v3`, `test_set_30k`) | n | KL | ≈ decisions to gather one nat (1 / KL) |
|
| 249 |
+
|---|---|---|---|
|
| 250 |
+
| `noul` | 8,537 | 0.005 | ≈ 200 |
|
| 251 |
+
| `choice` | 8,312 | 0.028 | ≈ 36 |
|
| 252 |
+
| `score` | 8,527 | 0.023 | ≈ 43 |
|
| 253 |
+
| **all Jev-labelled rows** | **25,376** | **≈ 0.019** | **≈ 54** |
|
| 254 |
|
| 255 |
+
JEV-27B reaches ≈ 0.017 (≈ 60 decisions per nat) on the same rows.
|
| 256 |
+
|
| 257 |
+
The other test rows are not labelled by Jev and are not part of this claim: Open-Jev rows carry programmatic ground
|
| 258 |
+
truth (in-distribution KL 0.004 for `noul`, 0.176 for `choice`; 0.234 on the OOD split of unseen task families), and
|
| 259 |
+
the `yuri_v1` rows carry placeholder labels. No Jev-labelled out-of-distribution set exists in the corpus, so the claim
|
| 260 |
+
is established on the 53 training domains; outside them, the independent benchmark with human labels (90–97 % of Jev's
|
| 261 |
+
accuracy) is the best available evidence.
|
| 262 |
+
|
| 263 |
+
**Fidelity includes the teacher's mistakes.** On a poker spot where a solver always checks, TypeSafe Jev 1.13 shoves
|
| 264 |
+
with 0.62 in a published test; JEV-9B shoves too, with 0.70 (JEV-27B 0.63). A faithful copy of System 1 is also a
|
| 265 |
+
faithful copy of its blind spots. At 9 B the student also adds some of its own: on an independent benchmark 11.5 % of
|
| 266 |
+
its 16-option answers change when only the option order changes, against 7.0 % for the teacher (JEV-27B 7.4 %).
|
| 267 |
+
|
| 268 |
+
## The Blocks of Experts recipe
|
| 269 |
+
|
| 270 |
+
```
|
| 271 |
+
┌── System 2 block: lm_head (248,320 × 4096) ───────► text generation and reasoning
|
| 272 |
+
Request ─► Router ─► Qwen3.5-9B backbone (frozen, bit-identical to the base) (adapter off; HumanEval 70.7 % = base)
|
| 273 |
+
per │
|
| 274 |
+
request └── + System 1 block: LoRA (40.1 M) + 24-slot head (98 k) ─► calibrated typed decision
|
| 275 |
+
(adapter on, decision path only) (one prefill pass; KL ≈ 0.019 to Jev)
|
| 276 |
+
```
|
| 277 |
+
|
| 278 |
+
| block | what it is | parameters | trained? | used for |
|
| 279 |
+
|---|---|---|---|---|
|
| 280 |
+
| Backbone | `Qwen/Qwen3.5-9B` text tower (vision tower and MTP head dropped), bf16 | 7.9 B | no — bit-identical to the base | both systems |
|
| 281 |
+
| **System 2 block** | the original `lm_head` (248,320 × 4096) | part of the base | no | text generation and step-by-step reasoning |
|
| 282 |
+
| **System 1 block** | LoRA r=16 on the decoder projections + a 24-slot fp32 decision head initialised from `lm_head` rows | 40.1 M + 98 k | yes, ≈ 3 B200-hours | calibrated typed decisions |
|
| 283 |
+
| Router | per request: the vLLM LoRA module `jev-decision`, or `peft` adapter on/off | — | — | chooses the system |
|
| 284 |
+
|
| 285 |
+
**Why separate blocks rather than one merged fine-tune.** Folding the System 1 LoRA into the backbone would let a
|
| 286 |
+
single weight set serve both heads, but it costs generation quality: the merged backbone with the original `lm_head`
|
| 287 |
+
scores 61.6 % (101/164) on HumanEval against 70.7 % for the base, a 9-point drop, even though prose perplexity barely
|
| 288 |
+
moves (3.15 → 3.30). Keeping the backbone pristine and applying the System 1 block only on the decision path removes
|
| 289 |
+
that trade-off. For decision serving the adapter is merged *in memory* at start-up, so decision latency matches a
|
| 290 |
+
merged bundle.
|
| 291 |
+
|
| 292 |
+
**Why the recipe is this efficient.**
|
| 293 |
+
|
| 294 |
+
1. **Pretraining does most of the work; distillation sharpens.** The decision head is initialised from the backbone's
|
| 295 |
+
own `lm_head` rows for the verbalizer tokens (`false/true`, `0`–`5`, `A`–`P`), so at step 0 its output equals the
|
| 296 |
+
pretrained model's zero-shot restricted next-token distribution (verified to |Δp| < 1e-5; measured 8.6e-07). Before
|
| 297 |
+
seeing a single label it already agrees with the test targets on 53 % of `choice` questions with `noul` AUROC 0.82;
|
| 298 |
+
distillation takes it to 90 % / 0.996.
|
| 299 |
+
2. **Small trainable footprint.** 40.2 M parameters — 0.5 % of the backbone. Validation KL was already below 0.10 after
|
| 300 |
+
the first 64 k rows, test KL reached 0.028 after 0.49 epoch (≈ 1.7 B200-hours) and 0.021 after 0.93 epoch.
|
| 301 |
+
3. **Transfer to unseen tasks.** The pretrained backbone reads the *content* of a new task instead of matching surface
|
| 302 |
+
patterns of the training domains: KL 0.234 and top-1 0.918 against the programmatic labels of the OOD split. It
|
| 303 |
+
also reads real, long, structured states (prose, JSON game states, policy documents; up to 856 tokens in the corpus).
|
| 304 |
+
4. **Reads options, not positions.** With 30 % option-permutation augmentation, the top-1 flip rate under shuffled
|
| 305 |
+
`choice` options is 3.9 %; the same backbone before distillation flips 38 % of the time.
|
| 306 |
+
5. **Calibration falls out of the objective.** Distilling full teacher distributions with KL (plus an ordinal RPS term
|
| 307 |
+
for `score`) gives fitted temperatures of 1.002 / 0.984 / 1.012 and ECE 0.0007 with no post-hoc correction.
|
| 308 |
+
6. **It scales without code changes, and scale pays off.** The same code, hyper-parameters and packaging produced the
|
| 309 |
+
second-generation JEV-27B; only `model_path` and memory settings changed (the head-initialisation identity holds
|
| 310 |
+
there too, 4.5e-07). Going from 9 B to 27 B lowers KL to Jev from ≈ 0.019 to ≈ 0.017, halves OOD KL
|
| 311 |
+
(0.234 → 0.104), and raises the System 2 path from 70.7 % to 78.0 % on HumanEval.
|
| 312 |
+
|
| 313 |
+
## Benchmark highlights
|
| 314 |
|
| 315 |
### Independent benchmark: side by side with TypeSafe Jev 1.13
|
| 316 |
|
| 317 |
[`gazelle93/decision-models-under-pressure`](https://github.com/gazelle93/decision-models-under-pressure) (published
|
| 318 |
25 Sep 2026) asks decision models to pick the right label for real texts from CLINC-150, MTOP, GoEmotions, DBpedia and
|
| 319 |
+
financial tweets under three kinds of pressure: more options, near-miss options, and shuffled option order. The labels
|
| 320 |
+
are human gold labels, none of this data is in our training set, and TypeSafe Jev 1.13's results are published with it.
|
| 321 |
+
We re-ran the same items with autotrust/JEV-9B and autotrust/JEV-27B, up to our 16-option limit.
|
| 322 |
|
| 323 |
| | TypeSafe Jev 1.13 (published) | **autotrust/JEV-9B** | autotrust/JEV-27B |
|
| 324 |
|---|---|---|---|
|
|
|
|
| 328 |
| Answers changed by shuffling the options alone (16 options, 5 orderings) | 7.0 % | **11.5 %** | 7.4 % |
|
| 329 |
| Time for 14,400 decisions on one B200 | — | **42 s** | 110 s |
|
| 330 |
|
| 331 |
+
On data it was never trained on, JEV-9B reaches 97 % of the teacher's accuracy with 2 and 4 options, 94 % with 8 and
|
| 332 |
+
90 % with 16: it falls behind faster than JEV-27B (96–98 %) as the option list grows, loses a little more on near-miss
|
| 333 |
+
options, and is more sensitive to option order than the teacher. Our run follows the benchmark's published method (gold
|
| 334 |
+
plus the first K−1 distractors of a pool, shuffled per item); the orderings are seeded differently, so compare
|
| 335 |
+
aggregates, not individual items.
|
|
|
|
| 336 |
|
| 337 |
### Fresh examples (Hacker News and V2EX, 23–25 September 2026)
|
| 338 |
|
| 339 |
+
Expected answers were written by hand before the model was run. These are illustrations (≈ 110 decisions), not a
|
| 340 |
benchmark.
|
| 341 |
|
| 342 |
| task | autotrust/JEV-9B | autotrust/JEV-27B |
|
|
|
|
| 350 |
|---|---|---|
|
| 351 |
| HN: "Two-tier encryption in the UK" | topic (10 options) | security & privacy · 0.87 |
|
| 352 |
| HN: "Using LLMs to trace alchemical knowledge and decode 17th century letters" | about AI? | P(true) = 0.88 |
|
|
|
|
| 353 |
| HN comment: "Please stop this. We've asked you before to observe the guidelines…" | what is it mainly doing? | moderating the discussion · 0.78 |
|
| 354 |
| V2EX: "一个不需要 gemini pro 的完全免费��注册 Muse 的方法 … 邀请码:…" | contains a referral / invite code? | P(true) = 1.00 |
|
| 355 |
| V2EX: "今天中秋节,还要加班的有吗?来报道下" | promotes a product or paid offer? | P(true) = 0.01 |
|
| 356 |
| Diff replacing a parameterised query with `"… WHERE id = " + request.args["id"]` | introduces a security vulnerability? | P(true) = 0.93 (0.17 for a variable rename) |
|
| 357 |
| "I'm not happy with the fit. What are my options here?" | asking for a refund? | P(true) = 0.17 (TypeSafe's docs report 0.22 for Jev on this exact text) |
|
| 358 |
|
| 359 |
+
Where it failed or wavered:
|
| 360 |
|
| 361 |
+
* **Code-rule check**: missed a TypeScript file that declares `const port: number = Number(process.env.PORT)` against
|
| 362 |
+
the rule "a port must be a branded, range-checked integer" (0.33); JEV-27B flags it (0.93).
|
| 363 |
+
* **Fraud screening**: a CEO wire-transfer (business-email-compromise) message is flagged, but only at 0.56 (JEV-27B 0.84).
|
| 364 |
* **Comment intent**: "Because it's not a real argument. It's a deflection people use." read as attacking another
|
| 365 |
commenter (0.66) rather than arguing a point.
|
| 366 |
+
* **Chinese promotion**: a V2EX post launching a paid HTTPS debugging tool was not flagged as promotional (0.44;
|
| 367 |
+
JEV-27B 0.77).
|
| 368 |
+
* **A poker spot with the nuts** (check or shove four times the pot; a solver checks 100 %): shoves with 0.70; the
|
| 369 |
+
teacher shoved with 0.62, so this mistake comes from the teacher.
|
| 370 |
+
* Counting ("more than 3 fruits?" / "more than 5?" for a list of 4: 0.84 / 0.37), date comparisons and an instruction
|
| 371 |
+
injected inside the state were handled correctly, but on a handful of examples only.
|
| 372 |
|
| 373 |
Per-example outputs and the benchmark aggregates are in `reports/realworld_9b.json` (the HN and V2EX inputs came from
|
| 374 |
their public APIs on 25 September 2026).
|
| 375 |
|
| 376 |
+
## Quickstart with vLLM (recommended)
|
| 377 |
|
| 378 |
+
**One vLLM engine serves both systems from the same pristine weights.** Ordinary requests go through the base
|
| 379 |
+
`lm_head` (System 2, exactly Qwen3.5-9B); requests addressed to the LoRA module `jev-decision` go through the decision
|
| 380 |
+
head (System 1). `adapter_vllm/` contains the backbone LoRA plus the 24-slot decision head re-expressed as an `lm_head`
|
| 381 |
+
LoRA (only the 24 verbalizer rows change), so a typed decision is a single prefill step with `max_tokens=1`,
|
| 382 |
+
constrained to the option tokens and read back as log-probabilities.
|
| 383 |
|
| 384 |
### 1 — Start the server (OpenAI-compatible)
|
| 385 |
|
| 386 |
```bash
|
| 387 |
+
hf download autotrust/JEV-9B --local-dir JEV-9B # ~18 GB
|
| 388 |
vllm serve JEV-9B --served-model-name autotrust/JEV-9B \
|
| 389 |
--enable-lora --max-lora-rank 32 --lora-modules jev-decision=JEV-9B/adapter_vllm \
|
| 390 |
--logprobs-mode processed_logprobs --max-model-len 4096
|
| 391 |
```
|
| 392 |
|
| 393 |
+
`--logprobs-mode processed_logprobs` is required: it makes the returned log-probabilities respect `allowed_token_ids`.
|
| 394 |
+
`--max-model-len 4096` is sized for decisions; raise it (for example to 16384) if System 2 requests will think at
|
| 395 |
+
length. Add `--enable-prefix-caching --mamba-cache-mode align` if you ask many questions about the same state.
|
| 396 |
|
| 397 |
+
### 2 — System 2: generation and reasoning (the unmodified base model)
|
| 398 |
|
| 399 |
```bash
|
| 400 |
curl -s localhost:8000/v1/chat/completions -H 'content-type: application/json' -d '{
|
|
|
|
| 403 |
"max_tokens": 60, "chat_template_kwargs": {"enable_thinking": false}}'
|
| 404 |
```
|
| 405 |
|
| 406 |
+
Set `"enable_thinking": true` for deliberate, step-by-step reasoning. This path is Qwen3.5-9B unchanged; see the
|
| 407 |
+
[Qwen3.5-9B model card](https://huggingface.co/Qwen/Qwen3.5-9B) for its reasoning benchmarks and recommended sampling
|
| 408 |
+
settings.
|
| 409 |
+
|
| 410 |
+
### 3 — System 1: typed decisions (Python, only `requests` + two small JSON files)
|
| 411 |
|
| 412 |
```python
|
| 413 |
import json, math, requests
|
|
|
|
| 439 |
"Is the customer asking for a refund?"))
|
| 440 |
```
|
| 441 |
|
| 442 |
+
Values can differ in the third decimal between runs: vLLM computes in bf16 and results depend slightly on which
|
| 443 |
+
requests are batched together. The head bias and the temperature are applied client-side; the log-softmax normaliser
|
| 444 |
that vLLM applies cancels out, so the result is exactly the decision head's calibrated distribution.
|
| 445 |
|
| 446 |
+
### 4 — System 1 → System 2: confidence-gated escalation
|
| 447 |
+
|
| 448 |
+
Because both systems live in one engine, a common pattern is to let System 1 answer when it is confident and hand the
|
| 449 |
+
rest to System 2. This is a usage pattern, not a configuration we have benchmarked; pick the threshold on your own
|
| 450 |
+
validation data, and serve with a `--max-model-len` large enough for the reasoning budget.
|
| 451 |
+
|
| 452 |
+
```python
|
| 453 |
+
def solve(state, question, options, threshold=0.90):
|
| 454 |
+
p = decide("choice", state, question, options) # System 1: one prefill pass
|
| 455 |
+
best = max(p, key=p.get)
|
| 456 |
+
if p[best] >= threshold:
|
| 457 |
+
return {"system": 1, "answer": best, "distribution": p}
|
| 458 |
+
prompt = (f"{state}\n\nQuestion: {question}\nOptions: " + "; ".join(options)
|
| 459 |
+
+ "\nThink it through, then give exactly one option on the last line.")
|
| 460 |
+
r = requests.post(f"{URL}/v1/chat/completions", json={ # System 2: same engine, base lm_head
|
| 461 |
+
"model": "autotrust/JEV-9B",
|
| 462 |
+
"messages": [{"role": "user", "content": prompt}],
|
| 463 |
+
"max_tokens": 8192, "chat_template_kwargs": {"enable_thinking": True}}).json()
|
| 464 |
+
return {"system": 2, "reply": r["choices"][0]["message"]["content"], "system1_distribution": p}
|
| 465 |
+
```
|
| 466 |
+
|
| 467 |
### Offline / batch (Python API)
|
| 468 |
|
| 469 |
```python
|
|
|
|
| 473 |
llm = LLM("JEV-9B", enable_lora=True, max_lora_rank=32, logprobs_mode="processed_logprobs", max_model_len=4096)
|
| 474 |
decision = LoRARequest("jev-decision", 1, "JEV-9B/adapter_vllm")
|
| 475 |
|
| 476 |
+
gen = llm.generate(["..."], SamplingParams(temperature=0.0, max_tokens=256)) # System 2, no LoRA
|
| 477 |
+
dec = llm.generate([prompt], [SamplingParams(max_tokens=1, temperature=1.0, # System 1
|
| 478 |
allowed_token_ids=ids, logprobs=len(ids))],
|
| 479 |
lora_request=decision) # then + bias, / T as above
|
| 480 |
```
|
| 481 |
|
| 482 |
+
Mixed batches work too: pass a per-request `lora_request` list (`None` for System 2, `decision` for System 1) and both
|
| 483 |
+
systems are served in the same `generate` call.
|
| 484 |
|
| 485 |
### Measured on one B200
|
| 486 |
|
| 487 |
| workload | PyTorch path | **vLLM** |
|
| 488 |
|---|---|---|
|
| 489 |
+
| System 2 — 164 HumanEval completions (greedy, ≤ 384 new tokens) | 165 s | **3.3 s** (≈ 50×) |
|
| 490 |
+
| System 1 — offline batch, 29,955 test questions | 75 s (398 q/s) | 80 s (374 q/s) |
|
| 491 |
+
| System 1 over HTTP — 64 / 256 concurrent clients | — | 150 / 205 req/s |
|
| 492 |
+
| System 1 fidelity vs. the PyTorch path | test KL 0.0210 | test KL 0.0211; mean \|Δp\| 0.0008 over HTTP |
|
| 493 |
| Many questions about one state, `--enable-prefix-caching` | — | +14–20 % throughput |
|
| 494 |
|
| 495 |
Notes:
|
| 496 |
+
* The big win is on System 2: generation is ≈ 50× faster. A decision is a single prefill pass with no decoding, so at
|
| 497 |
+
9 B offline batch throughput is about the same as the PyTorch path (at 27 B vLLM is 1.7× faster); for System 1 vLLM
|
| 498 |
+
mainly buys serving: continuous batching under concurrency, an OpenAI-compatible API, and one engine for both systems.
|
| 499 |
+
* Prefix caching: this architecture mixes Gated DeltaNet and attention layers, and vLLM caches it in blocks of 528
|
| 500 |
+
tokens, so only shared prefixes longer than 528 tokens are reused. The template puts `[kind]` before `[state]`, so
|
| 501 |
+
only questions of the same kind share a prefix. On 293 real states × 7.7 yes/no questions each (≈ 480-token states),
|
| 502 |
+
prefix caching served 19.6 % of prompt tokens from cache (+14–20 % throughput) with identical outputs.
|
|
|
|
| 503 |
* Requires a vLLM build with Qwen3.5 (`qwen3_5`) support, LoRA on `lm_head`, `--logprobs-mode` and
|
| 504 |
`allowed_token_ids`; tested with a vLLM development build from September 2026. Start-up takes 3–8 minutes
|
| 505 |
(CUDA-graph capture with LoRA enabled).
|
| 506 |
|
| 507 |
+
## What System 1 does
|
| 508 |
|
| 509 |
| kind | question | returns |
|
| 510 |
|---|---|---|
|
|
|
|
| 525 |
```
|
| 526 |
|
| 527 |
The template is tokenised as one string; the last token's final-norm hidden state goes through a
|
| 528 |
+
**linear fp32 head `H → 24 slots`** (`noul` → slots 0–1, `score` → 2–7, `choice` → 8–23). Inactive slots are masked,
|
| 529 |
+
a per-kind temperature is applied, and a softmax yields the distribution aligned with your `options`. One prefill
|
| 530 |
+
pass, no decoding.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 531 |
|
| 532 |
+
## Other ways to run it
|
| 533 |
|
| 534 |
+
### Plain `transformers` + `peft`
|
|
|
|
|
|
|
|
|
|
|
|
|
| 535 |
|
| 536 |
```python
|
| 537 |
import json, torch
|
|
|
|
| 544 |
tok = AutoTokenizer.from_pretrained(repo)
|
| 545 |
base = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="cuda") # == Qwen3.5-9B text model
|
| 546 |
|
| 547 |
+
# --- System 2: the pristine base model, no adapter ----------------------------------------------
|
| 548 |
msgs = [{"role": "user", "content": "In two sentences, what is safety stock?"}]
|
| 549 |
enc = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True).to("cuda")
|
| 550 |
out = base.generate(**enc, max_new_tokens=80)
|
| 551 |
print(tok.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True))
|
| 552 |
|
| 553 |
+
# --- System 1: attach the LoRA adapter (merged here for speed) + the 24-slot head ---------------
|
| 554 |
model = PeftModel.from_pretrained(base, repo, subfolder="adapter").merge_and_unload()
|
| 555 |
head = load_file(hf_hub_download(repo, "head.safetensors"))
|
| 556 |
cfg = json.load(open(hf_hub_download(repo, "judge_config.json")))
|
|
|
|
| 574 |
# {'false': 0.009, 'true': 0.991}
|
| 575 |
```
|
| 576 |
|
| 577 |
+
`options` are validated: `noul` must be `["false","true"]`, `score` must be `["0".."5"]`, `choice` takes 2–16
|
| 578 |
+
free-text options. Note that `merge_and_unload()` above changes the backbone for the rest of the process; to keep both
|
| 579 |
+
systems in one process, leave the adapter unmerged and run System 2 inside `with model.disable_adapter():`.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 580 |
|
| 581 |
## Evaluation details
|
| 582 |
|
| 583 |
+
### Additional System 1 metrics (`test_set_30k`, temperature applied)
|
| 584 |
|
| 585 |
| metric | autotrust/JEV-9B |
|
| 586 |
|---|---|
|
| 587 |
+
| `noul` Brier score against the target probability, all rows (lower is better) | 0.0015 |
|
| 588 |
| `score` ranked probability score (lower is better) | 0.0085 |
|
| 589 |
| Fitted temperatures noul / choice / score | 1.002 / 0.984 / 1.012 |
|
| 590 |
| Top-1 flip rate when `choice` options are shuffled (1,000 rows × 4 permutations) | 3.9 % |
|
| 591 |
| Out-of-distribution split — top-1 agreement · `noul` AUROC | 0.918 · 0.989 |
|
| 592 |
| Throughput — batch of 128 requests on one B200 | 2.5 ms per decision (≈ 400 decisions/s) |
|
| 593 |
+
| Single request on one B200 (median) | 87 ms |
|
| 594 |
|
| 595 |
+
### System 2 — no-degradation check (HumanEval, greedy pass@1, completion-style prompt)
|
| 596 |
|
| 597 |
| weights | pass@1 | note |
|
| 598 |
|---|---|---|
|
| 599 |
| Qwen3.5-9B (base) | 70.7 % (116/164) | same loader and protocol as below |
|
| 600 |
+
| **autotrust/JEV-9B — System 2 path (backbone + `lm_head`, adapter off)** | **70.7 % (116/164)** | all 164 completions byte-identical to the base model |
|
| 601 |
+
| System 1 LoRA folded into the backbone + base `lm_head` (*not shipped*) | 61.6 % (101/164) | why the blocks are kept separate |
|
| 602 |
|
| 603 |
### Per source × primitive (`test_set_30k`, temperature applied)
|
| 604 |
|
|
|
|
| 607 |
| yuri_v3 — synthetic operational scenarios, labelled by TypeSafe Jev 1.13 | noul | 8,537 | 0.005 | 0.961 | 0.001 | 0.994 | — |
|
| 608 |
| yuri_v3 | choice | 8,312 | 0.028 | 0.902 | 0.002 | — | — |
|
| 609 |
| yuri_v3 | score | 8,527 | 0.023 | 0.883 | 0.002 | — | 0.103 |
|
| 610 |
+
| openjev_v2 — Open-Jev programmatic tasks, ground-truth labels (not Jev) | noul | 1,432 | 0.004 | 0.998 | 0.003 | 1.000 | — |
|
| 611 |
| openjev_v2 | choice | 887 | 0.176 | 0.857 | 0.020 | — | — |
|
| 612 |
| yuri_v1 — placeholder `[0.5, 0.5]` labels (see Limitations) | noul | 2,260 | 0.000 | — | 0.005 | — | — |
|
| 613 |
|
| 614 |
+
OOD split (13,058 Open-Jev rows from task families not in training, programmatic labels): KL 0.234, top-1 0.918, noul
|
| 615 |
+
AUROC 0.989; `choice` KL 0.351 / top-1 0.837 (game-state decisions are the hardest slice).
|
| 616 |
|
| 617 |
+
Choice option-permutation consistency (1,000 rows × 4 random permutations): mean max |Δp| 0.024, p90 0.055, top-1
|
| 618 |
+
flip rate 3.9 %.
|
| 619 |
|
| 620 |
+
### Training trajectory (most recent first)
|
| 621 |
|
| 622 |
Validation KL on a fixed 4 k-row subset; `test_set_30k` metrics after calibration.
|
| 623 |
|
|
|
|
| 630 |
| step 500 | 64 k | 0.094 | 0.081 | 0.817 | 0.224 | 0.977 | 0.022 |
|
| 631 |
| untrained backbone with the initialised head (reference point, not the model) | 0 | 0.485 | 0.510 | 0.532 | 1.130 | 0.824 | 0.094 |
|
| 632 |
|
| 633 |
+
Annealing matters: the v0.7.0 cool-down (500 steps, lr ×0.9 → ×0.02 from step 2000) lowered KL by 25 %; continuing on
|
| 634 |
+
the unseen remainder of the epoch with the learning rate decayed to ≈ 0.07× (v0.8.0) lowered it by another 24 % and
|
| 635 |
+
added 1.4 points of choice agreement. JEV-27B folds this into a single cosine schedule.
|
| 636 |
|
| 637 |
## Training details
|
| 638 |
|
| 639 |
| item | value |
|
| 640 |
|---|---|
|
| 641 |
+
| teacher / data | `SargeDev/jev-distill-corpus-v3` (740,957 rows; `train` 655,806) with three streams: `yuri_v3` (498,010 rows, TypeSafe Jev 1.13 full output distributions via OpenRouter), `openjev_v2` (94,801 rows, Open-Jev programmatic labels, CC0), `yuri_v1` (148,154 rows, placeholder labels, down-weighted) |
|
| 642 |
| backbone | `Qwen/Qwen3.5-9B` text tower only (vision tower and MTP head dropped), bf16, frozen |
|
| 643 |
+
| System 1 block (trainable) | LoRA r=16, α=32, dropout 0.05 on `in_proj_qkv, in_proj_z, out_proj, q/k/v/o_proj, gate/up/down_proj` (40.1 M, shipped unmerged in `adapter/`) + 24-slot head (98 k, fp32, initialised from `lm_head` rows) |
|
| 644 |
+
| System 2 block | the original `lm_head`, not trained |
|
| 645 |
+
| loss | KL(target ‖ model) over active slots + 0.5 · RPS (ranked probability score) for `score` |
|
| 646 |
| augmentation | 30 % random permutation of `choice` options (targets permuted consistently) |
|
| 647 |
| batching | 128 rows / step, kind-stratified (≥ 1/6 per primitive), length-bucketed, micro-batches capped at 24 k padded tokens, gradient checkpointing |
|
| 648 |
| optimiser | AdamW (fused), β=(0.9, 0.98), lr head 2e-4 / LoRA 1e-4, cosine, warmup 3 %, grad-clip 1.0 for 2,500 steps; then continued on the unseen remainder of the epoch (fresh AdamW state, warmup 2 %, lr ×0.9 → cosine) and stopped after 2,250 more steps at lr ≈ ×0.07 — 4,750 steps ≈ 0.93 epoch in total |
|
|
|
|
| 653 |
|
| 654 |
## Limitations
|
| 655 |
|
| 656 |
+
* **System 1 mirrors TypeSafe Jev 1.13, including its mistakes.** This is a distillation, not an independent judge:
|
| 657 |
+
where the teacher was wrong or uncalibrated, so is autotrust/JEV-9B. Published evaluations of the teacher show it is
|
| 658 |
+
unreliable for multi-hop reasoning, arithmetic, dates, counting and adversarial inputs, and the student inherits all
|
| 659 |
+
of that. Confirmed on fresh inputs: the poker shove (0.70 vs the teacher's 0.62).
|
| 660 |
+
* **At 9 B it adds some blind spots of its own.** 11.5 % of its 16-option answers change with option order alone
|
| 661 |
+
(teacher 7.0 %), it keeps 90 % rather than 96 % of the teacher's accuracy at 16 options, and it missed a code-rule
|
| 662 |
+
violation that JEV-27B catches. Prefer [autotrust/JEV-27B](https://huggingface.co/autotrust/JEV-27B) for long option
|
| 663 |
+
lists, unfamiliar task families and code-rule checks.
|
| 664 |
+
* **The two systems are integrated in serving, not in knowledge.** System 1 cannot explain its decisions, and System 2
|
| 665 |
+
is the unmodified base model: it knows nothing about the decisions it is packaged with and was not trained to agree
|
| 666 |
+
with System 1. If you escalate from System 1 to System 2, expect them to disagree sometimes.
|
| 667 |
+
* **"Indistinguishable" is a KL statement on Jev-labelled rows from the 53 training domains.** The corpus has no
|
| 668 |
+
Jev-labelled out-of-distribution set; the OOD figures (KL 0.234, 0.351 for game-state `choice`) are measured against
|
| 669 |
+
programmatic ground truth, and on the independent benchmark the student reaches 90–97 % of Jev's accuracy, not 100 %.
|
| 670 |
+
* **Speed comparisons with the hosted API are not like for like.** Our timings exclude network time; the hosted
|
| 671 |
+
figures are third-party measurements that include it and depend on client concurrency and rate limits.
|
| 672 |
+
* **Choice agreement is capped by teacher ambiguity.** The teacher's `choice` distributions are soft (median top-1
|
| 673 |
+
probability 0.70). On the 14 % of rows where the teacher's top two options are within 0.1 of each other, argmax
|
| 674 |
+
agreement is near chance for *any* faithful mimic (0.46 where the gap is < 0.05). On teacher-decisive rows agreement
|
| 675 |
+
is 0.954, and the student's argmax captures 97.7 % of the teacher probability mass a perfect mimic could (0.693 vs
|
| 676 |
+
0.709).
|
| 677 |
+
* **Fixed option sets.** `noul` and `score` accept only their canonical options; `choice` accepts 2–16 options. Inputs
|
| 678 |
+
longer than 1,024 tokens are truncated (state only, head 60 % / tail 40 %) at serving unless you raise the limit.
|
| 679 |
+
* **English-centric.** The corpus is English; multilingual behaviour is inherited from the backbone and was not
|
| 680 |
+
systematically measured (the Chinese V2EX examples above are illustrations only).
|
| 681 |
+
* **Placeholder labels in the corpus.** The `yuri_v1` memory-relevance stream is 100 % exact-uniform `[0.5, 0.5]` —
|
| 682 |
+
those rows teach nothing about relevance. The model outputs ≈ 0.5 on them by design; do not use it for
|
| 683 |
+
memory-relevance scoring without further training.
|
| 684 |
+
* **Not for high-stakes decisions.** Use confidence gating: act automatically only above a threshold you validated on
|
| 685 |
+
your own data, and route the rest to System 2, a stronger model, or a human.
|
| 686 |
|
| 687 |
## Files
|
| 688 |
|
|
|
|
| 690 |
model-0000{1..5}-of-00005.safetensors Qwen3.5-9B text backbone incl. lm_head — bit-identical to the base model
|
| 691 |
(bf16; GDN A_log / gated-norm weights fp32 as in the original), 17.9 GB
|
| 692 |
model.safetensors.index.json · config.json
|
| 693 |
+
adapter/ System 1 LoRA (peft format, r=16, 40.1 M params, 154 MB) — apply only for decisions
|
| 694 |
head.safetensors 24-slot decision head (fp32): proj.weight [24, 4096], proj.bias [24]
|
| 695 |
judge_config.json slot layout, verbalizer token ids, template version, weights_mode=unmerged, provenance
|
| 696 |
calibration.json per-kind temperatures (+ fit diagnostics)
|
| 697 |
+
adapter_vllm/ the same adapter for vLLM: backbone LoRA (zero-padded to r=32) + decision head as an
|
| 698 |
+
lm_head LoRA, plus decision_head.json (head bias, verbalizer token ids)
|
| 699 |
tokenizer.json · tokenizer_config.json · chat_template.jinja
|
| 700 |
reports/ evaluation reports: test-set evaluation, bundle checks, HumanEval per-problem
|
| 701 |
results, vLLM measurements, real-world tests, training-milestone reviews
|
|
|
|
| 704 |
## License and acknowledgements
|
| 705 |
|
| 706 |
Weights: **Apache-2.0** (base model `Qwen/Qwen3.5-9B` is Apache-2.0; training corpus
|
| 707 |
+
`SargeDev/jev-distill-corpus-v3` is Apache-2.0, its `openjev_v2` stream additionally CC0). The System One framing and
|
| 708 |
+
the `noul` / `choice` / `score` primitives originate with TypeSafe AI's Jev; autotrust/JEV-9B is an independent
|
| 709 |
+
student model trained on public data and shares no weights, code or affiliation with TypeSafe AI.
|
| 710 |
|
| 711 |
```bibtex
|
| 712 |
+
@misc{autotrust_jev9b_2026,
|
| 713 |
+
title = {autotrust/JEV-9B: the first integrated System 1 + System 2 open model built with the Blocks of Experts recipe (Qwen3.5-9B; System 1 distilled from TypeSafe Jev 1.13)},
|
| 714 |
+
author = {{AutoTrust AI}},
|
| 715 |
year = {2026},
|
| 716 |
url = {https://huggingface.co/autotrust/JEV-9B}
|
| 717 |
}
|