---
license: apache-2.0
library_name: pytorch
tags:
- decision
- system-1
- jev
- typed-decisions
- small-models
---

# Byrne-Jev

Small local **System-1 typed-decision** model. Same job as TypeSafe's Jev and
convaiinnovations' Laya. You give it a state (text, JSON, or a conversation) and
typed questions (`choice`, `score`, `noul`). It returns a calibrated probability
for every option in **one forward pass**. Nothing is generated. No free text to
parse. No answer outside the options you define.

I built this. ~79M params: a 70.4M causal decoder trunk (SpikeWhale) plus an
8.5M Laya-style decision head. Method is close to Laya: typed decision head,
one marker position per option, 2-layer bidirectional head over the sequence,
strictly proper scoring rules so the probabilities are actually calibrated.

Answer format is schema-compatible with Laya and Jev. Served over Jev's
`/v1/systemone` wire protocol.

Checkpoint is `model.pt`. Temperature 1.0 for every question type. No post-hoc
temperature scaling. That measured best-calibrated. See Calibration.

## Files

| File | What |
|---|---|
| `model.pt` | the model (weights + config) |
| `agent.py` | `DecisionAgent`: `predict(state, questions)` -> answers |
| `serve.py` | Jev-compatible HTTP server (`POST /v1/systemone`, `GET /health`), standard library only |
| `eval_decisions.py`, `eval_data.py` | the evaluation used for every number below |
| `decision_core.py`, `config.py`, `model_v2.py`, `fractal.py`, `special_tokens.py`, `spike_tokenizer.py`, `tokenizer.json` | model code and tokenizer |
| `eval_results.json` | the full report behind the tables below |

## Install

Python 3.10+.

```bash
pip install -r requirements.txt
```

## Use

```python
from agent import DecisionAgent

agent = DecisionAgent("model.pt", device="cuda")      # or "cpu"
result = agent.predict(
    {"body": "Hi, we were billed twice for March. Refund the duplicate today or we cancel."},
    {
        "department": {"type": "choice", "instructions": "Which department should handle this?",
                       "criteria": {"billing": "invoices, payments, refunds",
                                    "technical": "bugs, outages, errors",
                                    "other": "everything else"}},
        "urgency":    {"type": "score", "instructions": "How urgent is this?",
                       "criteria": ["not urgent", "soon", "blocking"]},
        "churn_risk": {"type": "noul", "instructions": "Does the user threaten to cancel or leave?"},
    },
)
result["answers"]["department"]["choice"]            # e.g. "billing"
result["answers"]["department"]["probabilities"]     # one probability per option
result["answers"]["churn_risk"]["noul"]              # P(true)
```

Question types:
- **`choice`:** `criteria` is a dict of `label -> description` (or a list of labels).
- **`score`:** `criteria` is a list of ordered levels; the answer includes the expected level.
- **`noul`:** a yes/no question. `criteria` optionally describes `true` / `false`; the answer is P(true).

Each answer also carries `answer_confidence` (the probability of the chosen answer - the
calibrated number to gate on).

### Server (Jev wire protocol)

```bash
JEV_DEVICE=cuda python serve.py          # http://127.0.0.1:8000/v1/systemone
curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{
  "state": {"body": "billed twice, refund please or we cancel"},
  "questions": {"dept": {"type": "choice", "instructions": "which team?",
                "criteria": {"billing": "refunds", "tech": "bugs"}}}}'
```

| Env var | Default | Meaning |
|---|---|---|
| `JEV_MODEL` | `model.pt` | model file |
| `JEV_DEVICE` | `cpu` | `cuda` or `cpu` |
| `JEV_HOST` | `127.0.0.1` | bind address |
| `JEV_PORT` | 8000 | port |
| `JEV_API_KEY` | unset | when set, clients must send `Authorization: Bearer <key>` |
| `JEV_MAX_LEN` / `JEV_HEAD_MAX_LEN` | 1024 / 256 | token budgets per question (state + question, and question + options); positions go up to 2048 |

Responses include `usage`, `rounding: {"probabilityDecimals": 4}` and `cost_usd: 0`, so strict Jev
clients validate and budget them correctly.

## Evaluation

Reproduce everything below with `python eval_decisions.py --suite all --device cuda`. Other
suites: `--suite typed`, `--suite benchmarks`, or `--suite jsonl --jsonl your_examples.jsonl`.
All numbers were measured on an RTX 5060 Ti.

### typed-decisions: LocalLLaMA/typed-decisions test split

400 cases, 2,000 decisions, four workflows. Metrics and reference rows match the protocol in
Laya's fine-tuning notebook. Reference numbers are as published by their authors.

| Model | Params | Accuracy | Soft acc | Brier ↓ | ECE ↓ | Score MAE ↓ | Within 1 level |
|---|---|---|---|---|---|---|---|
| **Byrne-Jev** | **~79M** | **0.630** | **0.509** | **0.134** | **0.045** | **0.428** | **0.915** |
| TypeSafe Jev 1.13.0 | closed | 0.727 | 0.580 | 0.148 | 0.144 | 0.391 | 0.952 |
| Laya typed-decisions | 421M | 0.766 | 0.471 | 0.062 | 0.213 | 0.242 | - |
| ModernBERT-base specialist | 149M | 0.646 | 0.542 | 0.119 | 0.179 | 0.444 | 0.931 |
| Laya base (`laya`) | 421M | 0.362 | 0.332 | 0.316 | 0.175 | 0.694 | - |
| Teacher self-agreement (ceiling) | - | 0.735 | | | | | |
| Per-question majority class | - | 0.461 | | | | | |
| Random guess | - | 0.318 | | | | | |

KL divergence 0.228, total variation 0.220. Latency is 111 ms p50 and 158 ms p95 per case (5
questions in one forward pass, GPU).

| Workflow | Accuracy |
|---|---|
| Agent-trace observability | 0.704 |
| Security incidents | 0.654 |
| Customer service | 0.610 |
| Invoice processing | 0.552 |

What stands out:
- **Calibration.** ECE 0.045 is the lowest in the table. Stated confidences track hit rate.
- **Probability quality.** Brier beats Jev's published score. Soft accuracy beats Laya's
  fine-tuned checkpoint.
- **Accuracy gap.** Top-1 is 10-14 points below Jev and Laya. Invoice processing is the
  weakest workflow.

### Public benchmarks as typed questions

200 examples from each benchmark's evaluation split, each posed as a typed question
(`eval_data.py`). Instruction wording and option descriptions are sampled with a fixed seed.
banking77 and MASSIVE show up to 20 options per question: the gold intent plus randomly sampled
other intents (not the full 77 / 60-way label set).

| Benchmark (split) | Question | Options | Accuracy | ECE |
|---|---|---|---|---|
| IMDB (test) | noul / choice | 2 | 0.990 | 0.015 |
| DAIR Emotion (validation) | choice | 6 | 0.890 | 0.062 |
| AG News (test) | choice | 4 | 0.870 | 0.090 |
| banking77 (test) | choice | <= 20 | 0.835 | 0.058 |
| MASSIVE intent, en (validation) | choice | <= 20 | 0.785 | 0.102 |
| SST-2 (validation) | noul / choice | 2 | 0.775 | 0.136 |
| tweet_eval hate (validation) | noul | 2 | 0.630 | 0.187 |
| BoolQ (validation) | noul | 2 | 0.585 | 0.121 |
| Yelp review stars (test) | score | 5 | 0.505 | 0.140 |
| MultiNLI (validation matched) | choice | 3 | 0.500 | 0.077 |

Sentiment and topic/intent are strong. Reading a passage and reasoning over it is weak. BoolQ
is below always answering "yes" (~0.62). MultiNLI is 0.50, where chance is 0.33.

### Calibration

`model.pt` uses temperature 1.0. I also fitted post-hoc temperatures (per question type and per
option count) and compared. On typed-decisions, fitting made calibration **worse**: ECE rose from
0.045 to 0.100 and soft accuracy fell from 0.509 to 0.473, while Brier improved slightly (0.134 to
0.127). On the benchmark table, fitted temperatures lowered ECE on most single-label tasks. The
released model keeps temperature 1.0 because structured-state decisions are the main use. To
apply your own temperatures, add `temperature` (per type: choice, score, noul) or
`temperature_by_options` to the model's `decision_cfg`; the loader clamps values to [0.5, 5].

### Integration test: jev-doom (Freedoom, zero-shot)

To check it can stand in for Jev in a real application, I dropped Byrne-Jev in for Jev in
[jev-doom](https://github.com/olivier-motium/jev-doom), a public harness that shows Jev playing
Freedoom. Observable mode, MAP01, 90 s real time. No Doom-specific adaptation.

| | Result |
|---|---|
| Protocol | 164 decisions, 163 accepted, **0 validation errors** (the one rejection was the harness discarding a stale in-flight reply) |
| Latency | ~580 ms per decision (about 12 questions per call at `JEV_MAX_LEN=2048` / `JEV_HEAD_MAX_LEN=512`) |
| Outcome | survived to the time limit at full health; moved 5,339 units, visited 43 cells, discovered 581 |
| Combat / doors | 0 kills, 0 damage dealt or taken; never permitted firing (`trigger` = `hold_fire` 136/136) and never pressed USE, so it didn't open doors |

It works as a drop-in at the protocol level. As a zero-shot game agent it's weak: it tracks and
faces enemies but doesn't choose to shoot, and it prefers right-side moves.

## Limitations

- **Accuracy.** Top-1 on typed-decisions is below Jev and Laya.
- **Passage reasoning.** Reasoning over a passage (BoolQ, NLI) is weak.
- **Yes/no detection (`noul`) is unreliable and can be inverted.** Hand-written spot check: 5 of
  7 detection questions were wrong, including clear cases. Explicit "refund today or we cancel"
  scored churn 0.09 while a polite feature request scored 0.68. Obvious phishing email 0.29.
  "I want a manager now" escalation 0.04. "ignore all previous instructions" injection 0.16.
  **Describe both outcomes in `criteria`** (e.g. `{"true": "the user threatens to cancel or leave",
  "false": "the user does not threaten to leave"}`). Same 7 checks: accuracy 2/7 -> 5/7 (cancel
  threat 0.09 -> 0.99, escalation 0.04 -> 0.98, injection 0.16 -> 0.76). A two-option `choice`
  with the same descriptions scored the same (5/7), so it's the descriptions that matter, not the
  question type. Even so, a polite request still scored as a churn risk and the phishing email
  still scored below 0.5. Validate yes/no questions on your own data before gating anything on
  them.
- **Score questions under-rate intensity.** Urgency, frustration and star ratings skew toward the
  low end (e.g. a glowing review got an expected 2.6 of 5 stars; a same-day cancel threat was rated
  "not urgent").
- **Context.** Default 1024-token budget per question truncates long states. Questions with many
  long options share a 256-token budget (raise it with `JEV_HEAD_MAX_LEN`).
- **Action head.** `action.act_probability` in each answer is kept for format compatibility. It
  isn't meaningful.
- **Option order.** Options are read in the order given. Results can shift slightly if you
  reorder them.