--- license: apache-2.0 base_model: Qwen/Qwen3-0.6B-Base library_name: transformers pipeline_tag: zero-shot-classification tags: - assay - decision-model - calibrated - conformal-prediction - text-classification - structured-output language: - en --- # assay-0.6b Calibrated typed decisions from one forward pass. Send a state and named typed questions (`bool` yes/no, `choice` over 2..255 described options, `score` over 2..10 ordered levels); get a probability distribution per question, a confidence and an evidence score. No text is generated, so nothing can come back off-schema. Code, server and training recipe: https://github.com/bgokden/assay ## In one line **It abstains.** On tasks it has never seen, this model acts on 49% of bool questions at a 14.8% error rate, 39% of choice at 5.6%, and hands the rest over instead of guessing. That is a conformal prediction set against a fitted error rate, not a threshold on a softmax someone eyeballed. Accuracy on those unseen tasks is **0.704**. Every answer carries a calibrated probability, a confidence, and the set of options that could not be ruled out. The other five models, and the dataset, are in the [Assay collection](https://huggingface.co/collections/Berk/assay-calibrated-typed-decisions-6ab2fcb2b7eea0b7aaf785ab). ## How it is built - Backbone `Qwen/Qwen3-0.6B-Base` with a LoRA adapter (r=16, alpha=32, lr=5e-05, 1.0 epoch, batch 8 x 1 accumulation); merged weights are in this repository, the adapter is in `adapter/`. The merged weights load with transformers like any Qwen checkpoint. - The answer is read from the model's own next-token logits over option label tokens at a single decision position, so the base model's zero-shot competence is the starting point. - Questions are isolated branches over a shared state (block attention mask, restarted positions): packed and separate requests agree exactly. - Trained with cross-entropy against soft targets: human label distributions where the source has them, SORD-smoothed levels for ordinal questions, one-hot otherwise. Choice options are shuffled per example. - An evidence head (linear on the decision token, `assay_head.safetensors`) predicts whether the state supports the question, trained on passage-swapped negatives. - Global temperature 1.199 fitted on the calibration split of the training tasks and applied unchanged everywhere else. ## Evaluation | split | n | accuracy | Brier | NLL | ECE | confident errors | |---|---|---|---|---|---|---| | seen tasks (dev), raw | 6113 | 0.705 | 0.395 | 0.761 | 0.057 | 0.030 | | seen tasks (dev), scaled | 6113 | 0.705 | 0.391 | 0.744 | 0.030 | 0.019 | | unseen tasks (holdout), raw | 2020 | 0.704 | 0.402 | 0.732 | 0.052 | 0.031 | | unseen tasks (holdout), scaled | 2020 | 0.704 | 0.397 | 0.705 | 0.037 | 0.019 | | kev transfer-v4 dev, raw | 764 | 0.636 | 0.517 | 0.912 | 0.157 | 0.077 | | kev transfer-v4 dev, scaled | 764 | 0.636 | 0.499 | 0.851 | 0.124 | 0.045 | "Unseen tasks" are eleven datasets never used in training (bbc_news, app_reviews, scitail, medical_questions_pairs, tweet_irony, ethos, stance_climate, dream, copa, truthful_qa, hh_rlhf). "kev transfer-v4 dev" is the public suite from [jaredpalmer/kev-suites](https://huggingface.co/datasets/jaredpalmer/kev-suites) (mmlu, emotion, sciq, tweet_offensive, qnli, paws and synthetic rule holdouts); none of its sources are in the training data. Brier is the multi-class sum of squared errors (0..2), ECE uses 15 bins, confident errors are answers with p >= 0.9 that are wrong. | transfer-v4 source | n | accuracy | Brier | ECE | |---|---|---|---|---| | composition_held_and_or | 32 | 0.500 | 0.667 | 0.331 | | composition_held_conditional | 32 | 0.625 | 0.616 | 0.348 | | composition_held_or_not | 32 | 0.562 | 0.470 | 0.273 | | contrastive_authorization | 40 | 0.500 | 0.837 | 0.450 | | contrastive_deadline | 40 | 0.925 | 0.214 | 0.269 | | emotion | 116 | 0.457 | 0.708 | 0.235 | | mmlu | 116 | 0.457 | 0.636 | 0.168 | | paws | 80 | 0.588 | 0.548 | 0.266 | | qnli | 80 | 0.775 | 0.313 | 0.107 | | sciq | 116 | 0.888 | 0.193 | 0.068 | | tweet_offensive | 80 | 0.713 | 0.445 | 0.177 | Latency on one RTX 5090 (bf16, transformers, packed questions over one state versus separate requests): ``` questions packed_ms separate_ms 1 17.4 17.2 3 18.1 52.2 6 20.9 104.1 12 18.8 207.6 24 21.8 416.7 ``` ## Abstention `conformal.json` holds per-question-type thresholds fitted on the seen-task calibration split (alpha 0.1, delta 0.05): a prediction-set threshold with coverage at least 1 - alpha and an act threshold on the top probability whose acted-on error rate is at most alpha at confidence 1 - delta, both on inputs distributed like the calibration split. `assay.server` returns them as `act` and `set` on every answer. How they carry over to other tasks: | split | type | coverage | set size | act rate / error among acted | |---|---|---|---|---| | unseen tasks | bool | 0.88 | 1.28 | 49% / 14.8% | | unseen tasks | choice | 0.92 | 1.80 | 39% / 5.6% | | unseen tasks | score | 0.77 | 2.37 | no threshold | | transfer suite | bool | 0.74 | 1.17 | 68% / 31.2% | | transfer suite | choice | 0.87 | 1.91 | 36% / 15.8% | | transfer suite | score | 1.00 | 1.75 | no threshold | Score questions get prediction sets but usually no act threshold, because exact-level accuracy is the wrong error notion for ordinal answers. Refit on your own labelled data with `python -m assay.conformal` for a guarantee about your distribution. ## The family | model | size | unseen tasks | transfer-v4 | |---|---|---|---| | [assay-0.6b](https://huggingface.co/Berk/assay-0.6b) | 0.6B | 0.704 / 0.397 | 0.636 / 0.499 | | [assay-1.7b](https://huggingface.co/Berk/assay-1.7b) | 1.7B | 0.752 / 0.334 | 0.670 / 0.436 | | [assay-4b](https://huggingface.co/Berk/assay-4b) | 4B | 0.803 / 0.271 | 0.784 / 0.302 | | [assay-8b](https://huggingface.co/Berk/assay-8b) | 8B | 0.808 / 0.256 | 0.818 / 0.267 | | [assay-27b](https://huggingface.co/Berk/assay-27b) | 27B | 0.842 / 0.221 | 0.842 / 0.229 | | [assay-compiled-base](https://huggingface.co/Berk/assay-compiled-base) | 149M | 0.606 / 0.494 | 0.542 / 0.572 | Accuracy / Brier after temperature scaling. Same recipe, same splits, different backbones; per-tier abstention and latency are in [docs/models.md](https://github.com/bgokden/assay/blob/main/docs/models.md). ## Serving ```bash python -m assay.server --model Berk/assay-0.6b --port 8000 ``` `POST /v1/decide` is the native shape; `POST /v1/systemone` and `/v1/systemone/batch` accept the shape other open decision models use (`criteria` options, `noul` booleans); `POST /v1/decide_graph` walks a decision tree in one forward pass; `POST /v1/agents` registers an agent -- a graph plus the actions its outcomes stand for -- and `/v1/agents//run` decides a case. Requests arriving together share a pass, and `/health` and `/metrics` are for operations. `assay.backends.sglang` runs the same model on an SGLang deployment. Guides: [deployment and the full API](https://github.com/bgokden/assay/blob/main/docs/serving.md), [agents](https://github.com/bgokden/assay/blob/main/docs/agents.md), [runnable examples](https://github.com/bgokden/assay/tree/main/examples). ## Train one on your own data `python -m assay.pipeline --config .json` runs training, temperature calibration, evaluation and the conformal thresholds over your own records, and writes a directory this same server and publisher accept. The repository's [examples/](https://github.com/bgokden/assay/tree/main/examples) has a configuration per tier and a dataset in the record format; records written for other decision models (`criteria` options, `noul` booleans) load unchanged. ## Usage ```python from assay import load_model from assay.schema import Question model = load_model("Berk/assay-0.6b") answers = model.answer( state="My card was charged twice for order A-104.", questions={ "refund": Question(type="bool", instructions="Does the customer ask for money back?"), "team": Question(type="choice", instructions="Which team should handle this?", options={"billing": "Charges and refunds", "technical": "Bugs"}), }, ) print(answers["team"].probabilities, answers["refund"].p_true, answers["refund"].evidence) ``` ## Limitations Text only, English training data. No arithmetic, counting, date comparison or multi-hop reasoning in one pass; keep those in code. Accuracy drops with unrelated state. The evidence head is trained on coarse swapped-passage negatives. Probabilities are calibrated in aggregate on the evaluated distributions, which is not a guarantee about any single answer or about your data; check calibration on your own labels before acting on thresholds. ## Training data Fifty-five public classification, inference, reading-comprehension and preference datasets rendered as typed questions with described options, plus a synthetic policy-application generator (see `assay/data/tasks.py` in the repository for the rubrics). Each dataset keeps its own licence; the per-dataset list is in [docs/datasets.md](https://github.com/bgokden/assay/blob/main/docs/datasets.md). Several sources carry non-commercial or research-only terms; check them before commercial use. ## Relationship to other work Assay is an independent project. Jev and System One are names of TypeSafe AI's products and are mentioned only to describe and compare; kev-suites is Jared Palmer's evaluation data. Assay is not affiliated with or endorsed by either.