DecisionBench and Bosun v3.1: building and testing decision models

Today we are releasing DecisionBench, two Bosun v3.1 models (0.6B and 1.7B), and a Jev-compatible server. Together, they let you test decision models across different tasks, run an open-weight model that returns complete answer distributions, and change the model behind a familiar API.

You can try the runtime with a registered Transformers model:

uvx --from 'jev-compatible-server[transformers]' \
  jev-compatible-server --model bosun-v3.1-0.6b

Send a request to http://127.0.0.1:8000/v1/systemone. The --model flag pins this server to Bosun v3.1 0.6B, so the request does not need a model name. Later in this post, we will walk through that request and the logits behind its answer.

There are three pieces to understand:

First, what counts as a decision?

Many application calls have a bounded answer space. A routing step needs one tool from a list. An alert needs a yes-or-no judgment. A review might need an ordered score. Generating a paragraph and parsing it afterward is possible, but it leaves the application to recover the answer and its confidence from text.

Jev made a cleaner interface concrete: provide shared state, ask a typed noul, choice, or score question, and get an answer with probabilities. Those types correspond to binary classification, candidate selection, and ordinal scoring. The allowed choices can be supplied when the request is made.

Once we had that interface, we needed a way to measure the model behind it. A large number of near-duplicate rows can make a test precise on one task without telling us much about other decisions. We wanted the tasks themselves to remain visible.

DecisionBench: keep the task as the unit of evidence

We took inspiration from MTEB. Its leaderboard is useful because the matrix beneath it keeps retrieval, classification, clustering, languages, and datasets separate. The community can add a task, adapter, result, or fix without the benchmark belonging to today's leading model.

DecisionBench 1.0 begins with 43 tasks, 23,900 English rows, and 28 domains. It has three decision primitives. The standard applied suite has 22,700 rows; a separate 1,200-row reasoning track tests multistep, numerical, and evidence-heavy decisions. Both belong to the first release, but we do not blend them into one applied headline score. The dataset revision is pinned.

The primary score is all-row accuracy. An unsupported question or failed response counts as a miss. We also report coverage, expected calibration error (ECE), negative log likelihood (NLL), and results by task, family, domain, primitive, and candidate count. Accuracy asks whether the top choice is right. ECE and NLL tell us more about the probabilities the application may actually use.

Each official run preserves exact inputs, complete candidate distributions, raw responses, failures, revisions, and content-addressed artifacts. The benchmark repository holds task definitions, adapters, and evaluation code; the dataset holds frozen rows; the results repository holds reviewed records; and the leaderboard presents them. Adding a task does not silently rewrite a published benchmark.

That structure is meant to make DecisionBench a place to contribute evidence. You can add a model, task, benchmark, or result, including one that changes the order of the board.

Bosun: start with token probabilities

The design thread for Bosun comes from Qwen3's reranking work. In the Qwen3-Reranker example, the model is asked whether a document meets a query's requirements. The readout takes the final-position logits for the yes and no tokens and normalizes just those two values into a score. Our earlier Bosun reranker used the same kind of yes/no readout.

We liked the directness of that approach. The application receives a value derived from the model's answer-token probabilities rather than generating prose, locating an answer inside it, and then trying to assign confidence.

Bosun v3.1 extends the readout from two fixed tokens to a runtime answer space. It uses the ordinary Qwen3-0.6B or Qwen3-1.7B causal base, a rank-16 LoRA adapter, and 256 learned decision tokens. Up to 255 candidates can be presented in one question; one token is reserved for a null slot. These model packages contain the adapter, new token embeddings, tokenizer, model code, and pinned base-model details: Bosun v3.1 0.6B and Bosun v3.1 1.7B.

Here is the readout for one question, in execution order:

1. Put the state, instructions, decision type, and candidate descriptions into a structured prompt. Assign each presented candidate a decision-token slot.
2. Run the Qwen3-based model once and read the final-position logits for the decision tokens.
3. Mask tokens that were not presented, apply softmax over the valid slots, and map those probabilities back to the caller's candidate order.

A token ID is a slot, not a permanent label for a real-world action. Candidate presentation varies during training so that the model cannot rely on a particular token always meaning, for example, “rollback.” The released serving contract records the token and slot rules.

Training also targets a distribution over valid answers, rather than only a generated answer string. The model card describes 80,000 base decision examples plus 50,000 examples from additional decision-task families. The same readout handles yes or no, a choice among supplied candidates, and an ordered score. This is an extension of the Qwen3-Reranker probability idea, not a fine-tune of its reranker checkpoint.

What the first evaluation shows

On the 22,700-row standard applied suite, Bosun v3.1 occupies the first two positions on the initial leaderboard:

These are held-out rows from broad task families represented in Bosun's training data. They do not establish performance on wholly unseen task families. On the separate reasoning track, Jev leads both Bosun models. Bosun's training data is not part of this release. The task-level results and that reasoning result are part of the record alongside the applied wins.

A stable API for trying another implementation

The Jev-compatible server exposes POST /v1/systemone. A request supplies shared state and defines one or more typed questions. Here is a complete choice request for the server started above:

{
  "state": {
    "incident": "Checkout errors rose after the latest deployment"
  },
  "questions": {
    "next_action": {
      "type": "choice",
      "instructions": "Choose the next operational action.",
      "criteria": {
        "page": "Page the on-call engineer",
        "watch": "Continue monitoring",
        "rollback": "Roll back the deployment"
      }
    }
  }
}

Save that JSON as request.json, then send it to the server started above:

curl -X POST http://127.0.0.1:8000/v1/systemone \
  -H 'Content-Type: application/json' \
  --data @request.json

The response keeps the question name and returns a typed answer with candidate probabilities. With --model, the server pins one implementation; restart it with --model bosun-v3.1-1.7b to send the same request to the larger Bosun model. The optional request-level model field selects an alias when the server is running in registry mode. The registry records the backend, checkpoint or endpoint, revision, and readout configuration for each implementation. The server loads and caches runtimes by alias, then dispatches the same request shape to the selected adapter. It currently includes Transformers and llama.cpp paths; other integrations are marked pending.

Swapping the pinned model preserves the request shape. It does not promise that two models support every question type or return equally useful probabilities. The server reports unsupported questions explicitly, and DecisionBench gives you a public place to compare accuracy, calibration, coverage, and failures before testing on your own application data. The API reference documents the full request and response contract.

Help build the next DecisionBench

The first 43 tasks are a starting point. We need people who know the decisions this release misses, and people with models that might expose where the current leaders are weak. Bring us a task from your field, a model we have not run, or a result that changes the story. We want the board to get harder and more useful as the community builds it.

There are four ways to contribute:

• 🤖 Add a Model → Give another decision model a compatibility adapter so it can be evaluated on the same frozen rows.
• 📊 Submit Results → Run a supported model and submit its reproducible scores, complete outputs, and failures for review.
• 🧩 Add a Task → Bring a dataset-backed decision problem from a domain you know, with labels, provenance, and tests.
• 🗂️ Add a Benchmark → Assemble existing tasks into a named evaluation that answers a question the current board cannot.

If you find a case where Bosun fails, we especially want to see it. Add the task, submit the result, or open the issue. The benchmark improves when that failure becomes a test everyone can run.