README.md

---
license: apache-2.0
base_model: bespokelabs/Bespoke-Nimble-9B
library_name: peft
language:
- en
tags:
- lora
- classification
- agents
- decision
- nimble
---

# autonoxis-conductor-9b

A LoRA adapter for [bespokelabs/Bespoke-Nimble-9B](https://huggingface.co/bespokelabs/Bespoke-Nimble-9B)
(revision `594dfdcfb6f94e3d0c0db7535180d3c71689169a`) that makes "conductor" decisions for autonomous
coding agents. It does not generate text. Nimble scores the answer tokens and the adapter returns a
probability for each allowed label.

Two tracks, one question per request:

| track | labels |
|---|---|
| `decision`: what should the conductor do next with this lane? | `STOP`, `ASK`, `DISPATCH` |
| `manager`: what should happen to the evidence returned for this lane? | `ACCEPT`, `VERIFY`, `REJECT`, `REOPEN`, `ESCALATE` |

The question wording and the label criteria the adapter was trained on are in `conductor-questions.json`.

## How to use

You need the Nimble prompt builder and scorer from [github.com/bespokelabsai/nimble](https://github.com/bespokelabsai/nimble):
`nimble.scoring.parallel_schema.prepare_prompts` and `nimble.training.schema_train.candidate_logits`.

The call convention matches training. Use it exactly:

- ask **one** question per request, with the field id `label`;
- the state is the JSON `{"packet": <packet text>}`;
- the field's description and choice descriptions come from `conductor-questions.json` (`instructions` and `criteria` for that track).

Load the base model and attach the adapter **unmerged**. `merge_and_unload()` in bf16 shifts the
probabilities: on one evaluation packet VERIFY went from 0.7278 unmerged to 0.6765 merged.

```python
import json, torch
from transformers import AutoTokenizer, AutoModelForImageTextToText
from peft import PeftModel
from huggingface_hub import hf_hub_download
from nimble.scoring.parallel_schema import prepare_prompts
from nimble.training.schema_train import candidate_logits

BASE, REV = "bespokelabs/Bespoke-Nimble-9B", "594dfdcfb6f94e3d0c0db7535180d3c71689169a"
ADAPTER = "damianborek/autonoxis-conductor-9b"

tok = AutoTokenizer.from_pretrained(ADAPTER)
model = AutoModelForImageTextToText.from_pretrained(BASE, revision=REV, dtype=torch.bfloat16).to("cuda")
model = PeftModel.from_pretrained(model, ADAPTER).eval()   # do not merge
questions = json.load(open(hf_hub_download(ADAPTER, "conductor-questions.json")))

def ask(packet: str, track: str) -> dict:
    q = questions[track]
    field = {"type": "enum", "description": q["instructions"],
             "choices": list(q["criteria"]), "choice_descriptions": q["criteria"]}
    p = prepare_prompts(tok, json.dumps({"packet": packet}, ensure_ascii=False), {"label": field}, 2048)
    ids, cands = p.full_ids[0], p.candidate_ids[0]
    batch = {"input_ids": torch.tensor([ids], device="cuda"),
             "attention_mask": torch.ones(1, len(ids), dtype=torch.long, device="cuda"),
             "candidate_ids": torch.tensor([cands], device="cuda"),
             "candidate_mask": torch.ones(1, len(cands), dtype=torch.bool, device="cuda")}
    with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
        probs = candidate_logits(model, batch)[0].double().softmax(-1).tolist()
    return dict(zip(q["criteria"], probs))

print(ask("Lane 3 returned a passing test run for commit abc123, but the lane head is now def456 ...", "manager"))
```

**Confidence gate.** Compute `conf = (n * max_prob - 1) / (n - 1)`, where `n` is the number of labels.
Act on the top label only when `conf >= 0.8`. Below that, escalate to a stronger model or a human.

A local HTTP server with a Jev-compatible request shape (`autonoxis-server`) exists and loads the adapter
unmerged in the same way. It is not part of this repository, and there is no public hosted endpoint.

## Training

- Recipe: LoRA r=16, alpha=32, dropout 0.05, on the language-model Linear layers (attention, linear-attention
  and MLP projections). Candidate cross-entropy over the answer tokens. lr 5e-5, effective batch 8, 3 epochs,
  linear schedule with 10% warmup, bf16.
- Data: 1076 rows. 316 come from lab history. The other 760 are drafted contrastive rows aimed at the hard
  boundaries (ASK/STOP, REJECT/ESCALATE, REJECT/REOPEN, VERIFY/REJECT). Label disagreements were adjudicated
  by Fable 5 (`claude-fable-5`). The 760 drafted rows are disjoint from every evaluation set (checked at
  build time).
- This adapter is seed 1 of a 10-seed run. It was chosen by a fixed rule that looks only at v5 and the
  all-seed ensemble.
- The training data is not released.

## Results

The gold labels are Fable 5 (`claude-fable-5`). "Unsafe" means the model predicted `DISPATCH` or `ACCEPT`
where the gold label is different. The evaluation sets are not released. `eval-summary.json` holds the
per-set metrics for this adapter (no packet text).

**v8 (60 rows) is the fair check.** It was never used for training or selection.

| | v8 |
|---|---|
| 10 seeds, mean ± sd | 59.2 ± 0.9 / 60 (min 58, max 60) |
| this adapter (seed 1) | 58 / 60 (decision 24/24, manager 34/36) |
| unsafe | 0 in every seed |
| this adapter, conf ≥ 0.8 | keeps 59, accuracy 0.983 |

This adapter misses two v8 packets: `v8_g04` REJECT→VERIFY (conf 0.66, below the gate) and `v8_g27`
ACCEPT→VERIFY (conf 0.998).

v8 is **in-distribution**. It shares 7 scenario families with the training data and came from the same
drafting pipeline. It is not an out-of-distribution test.

**v5 to v7 are not clean held-out sets.** Targeted training batches were written from the contract rules
the model misapplied on these sets. The scenarios are new and none of the eval wording was reused, but the
gains are partly driven by those errors. Reported for completeness only:

| | v5 | v6 | v7 |
|---|---|---|---|
| 10 seeds, mean ± sd | 39.9 ± 0.3 / 40 | 47.4 ± 0.5 / 48 | 47.9 ± 0.3 / 48 |
| this adapter | 40 / 40 | 48 / 48 | 48 / 48 |

**References on the same packets:**

- Untrained Jev (TypeSafe) also scores 60/60 on v8. This adapter does not beat Jev. Its value is that it
  runs locally and costs nothing per call. In a 3-request check on one local GPU, the two requests after
  warm-up took 97 and 111 ms each (the first request, which included CUDA warm-up, took 494 ms).
- Stock Bespoke-Nimble-9B without the adapter: v6 41/48 and v7 40/48, with 11 confident-but-wrong answers
  at conf ≥ 0.8.
- To check the labels, Opus 5.5 (`claude-opus-5-5`) re-labelled all 196 v5–v8 packets blind. It agreed with
  the Fable 5 gold on 194 of 196.

## Limitations

- Prompts are limited to 2048 tokens. Longer packets are rejected, not truncated.
- English only.
- Narrow domain: conductor and manager decisions for autonomous coding lanes, as defined by one decision
  contract. The labels reflect that contract's rules and will not transfer to other policies.
- The probabilities are not calibrated beyond the evaluation sets above. The 0.8 gate was only checked on
  those sets.
- The adapter is trained for the single-question convention above. Asking both questions in one schema
  changes some predictions.

## License

Apache-2.0, the same as the base model. Bespoke-Nimble-9B is itself a LoRA-merged Qwen/Qwen3.5-9B, which is
also Apache-2.0.