--- license: other license_name: lfm1.0 license_link: LICENSE base_model: - LiquidAI/LFM2.5-VL-1.6B base_model_relation: finetune pipeline_tag: image-text-to-text library_name: transformers language: - en tags: - calibration - classification - system-one - jev-compatible datasets: - allenai/ai2_arc - allenai/sciq - allenai/openbookqa - tau/commonsense_qa - uoft-cs/cifar10 - mteb/stsbenchmark-sts ---
alpha-sys-1 banner
docs • nullsilver.com
# alpha-sys-1-1.6B alpha-sys-1 is a multimodal, Jev-compatible **System One model**. It takes a state, which may contain text, an image, or both, together with a question that has a fixed set of answers, and returns a probability distribution over those answers in one forward pass. It generates no text. The model is trained for calibrated probabilities: across a large group of similar examples where it assigns an answer a probability near 80%, that answer should be correct in roughly 80% of cases. Calibration degrades when the input differs substantially from the training data, so check the probabilities on data from the intended application. | | | |---|---| | Base | [LiquidAI/LFM2.5-VL-1.6B](https://huggingface.co/LiquidAI/LFM2.5-VL-1.6B) | | Tuning | LoRA rank 32, lr 1e-4, merged into the base weights | | Checkpoint | `alpha-sys-1-260919`, revision `260919`, seed 1 of 3 | | Input | text, one image, or both; English | | Output | probabilities over the answer space | | Sizes | [450M](https://huggingface.co/nullsilver/alpha-sys-1-450M) · [1.6B](https://huggingface.co/nullsilver/alpha-sys-1-1.6B) · [3B](https://huggingface.co/nullsilver/alpha-sys-1-3B) | ## Question types Questions follow TypeSafe's System One format: a request contains one `state` and any number of `questions`, so a question written for Jev runs here as is. `images` is an extra field for multimodal inputs. | type | answer space | returns | |---|---|---| | `choice` | named options, up to 26 | `probabilities` over the options, `choice` (argmax) | | `noul` | a statement | `noul`, P(true) | | `score` | ordered levels, lowest first | `probabilities` over the levels, `score` (expected level index) | The answer is read from the next-token logits for the answer labels (`A`, `B`, … or `No`/`Yes`), renormalised over the valid labels. Each question is answered independently: one question's answer is never context for another. > [!NOTE] > `confidence` is `1 - H(p)/log(n)`, computed from the distribution. It is not a separate > prediction. > [!TIP] > A `noul` probability near 0.5 means the model is uncertain. ## Usage `alpha_sys_1.py` in this repository renders questions the way the model was trained on them, batches the questions on one state, and returns answers in the System One shape. ```python from huggingface_hub import hf_hub_download import importlib.util, sys spec = importlib.util.spec_from_file_location("alpha_sys_1", hf_hub_download("nullsilver/alpha-sys-1-1.6B", "alpha_sys_1.py", revision="260919")) alpha_sys_1 = importlib.util.module_from_spec(spec); spec.loader.exec_module(alpha_sys_1) m = alpha_sys_1.SystemOne("nullsilver/alpha-sys-1-1.6B", revision="260919") m.system_one({ "state": {"subject": "Duplicate charge on invoice #4411", "body": "We were billed twice for March. Refund the duplicate today or we cancel our plan."}, "questions": { "department": {"type": "choice", "instructions": "Which department should handle this email?", "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages", "sales": "pricing, new contracts", "other": "everything else"}}, "urgency": {"type": "score", "instructions": "How urgent is this request?", "criteria": ["not urgent", "soon", "critical deadline or blocking issue"]}, "churn_risk": {"type": "noul", "instructions": "The user threatens to cancel or leave."}}}) # {"department": {"choice": "billing", "probabilities": {"billing": 0.98, "technical": 0.00, "sales": 0.01, "other": 0.01}, "confidence": 0.92}, # "urgency": {"score": 1.60, "probabilities": [0.10, 0.20, 0.70], ...}, # "churn_risk": {"noul": 0.59}} ``` Email triage is not one of the training environments; the output above is what this checkpoint returns on it, not a tuned result. Without the client, use this prompt format. A different format gives less reliable probabilities. ```python import string, torch from transformers import AutoModelForImageTextToText, AutoProcessor repo, rev = "nullsilver/alpha-sys-1-1.6B", "260919" processor = AutoProcessor.from_pretrained(repo, revision=rev) processor.tokenizer.padding_side = "left" model = AutoModelForImageTextToText.from_pretrained( repo, revision=rev, dtype=torch.bfloat16, device_map="auto").eval() def render(state, q): parts = [state] if state else [] if q["type"] == "noul": c = q.get("criteria") or {} clar = "".join(f"\n{lab} means: {c[k]}" for lab, k in (("Yes", "true"), ("No", "false")) if c.get(k)) parts.append(f"Statement: {q['instructions']}{clar}\nIs the statement true? Answer with Yes or No only.") return "\n\n".join(parts), ["No", "Yes"] crit = q["criteria"] items = list(crit.items()) if isinstance(crit, dict) else [(o, None) for o in crit] labels = list(string.ascii_uppercase[:len(items)]) lines = [f"{lab}. {o}" + (f": {d}" if d else "") for lab, (o, d) in zip(labels, items)] parts.append(q["instructions"] + "\n" + "\n".join(lines) + "\nAnswer with the letter only.") return "\n\n".join(parts), labels @torch.inference_mode() def ask(q, state="", image=None): text, labels = render(state, q) content = ([{"type": "image", "image": image}] if image is not None else []) + [{"type": "text", "text": text}] inputs = processor.apply_chat_template( [[{"role": "user", "content": content}]], add_generation_prompt=True, tokenize=True, return_dict=True, processor_kwargs={"return_tensors": "pt"}).to(model.device) logits = model(**inputs, logits_to_keep=1).logits[0, -1].float() ids = [processor.tokenizer.encode(lab, add_special_tokens=False)[0] for lab in labels] return torch.softmax(logits[ids], -1).tolist() p = ask({"type": "noul", "instructions": "The message conveys urgency"}, state="Our API integration started returning 500 errors an hour before launch.") urgent = p[1] # P(Yes) ``` > [!NOTE] > - A dict `state` is rendered one field per line, as `key: value`. > - Training images smaller than 256 px were upscaled to 256 px. > - For several questions on one state, batch them with `padding_side="left"`. > - In bfloat16, probabilities move by up to a few hundredths with batch composition and > padding. ## Training Training uses cross-entropy between the model's distribution and a target `y_soft`. The target is one-hot when a dataset provides one answer, and the annotator distribution when several annotations are available. Options are shuffled on every draw, the vision tower is frozen, and environments are sampled in proportion to the square root of their size. | environment | modality | type | label | |---|---|---|---| | mcq (ARC-Easy, SciQ, OpenBookQA, CommonsenseQA) | text | choice | one-hot | | ChaosNLI (100-annotator items) | text | choice | annotator distribution | | CivilComments-WILDS | text | noul | annotator share | | STS-B | text | score | annotator mean | | Folktables (ACS income, California 2014) | tabular as text | noul | outcome | | CIFAR-10 | image | choice | one-hot | | Camelyon17-WILDS | image | noul | outcome | Three random seeds were trained. The released checkpoint is the seed with the lowest mean development loss across environments. ## Evaluation Each test split was read once per checkpoint. Reported intervals are 95% clustered bootstrap intervals, clustered on the relevant dataset group: question, comment, hospital, or state-year. > [!IMPORTANT] > Compare models on NLL and Brier score. ECE is reported alongside them and is misleading > on its own: a model that always predicts the base rate can have a low ECE. The tables carry two reference points. The **base rate** is the constant predictor: it answers every question with the label frequencies of the training split (for example "toxic" 14% of the time on CivilComments, whatever the comment says), or uniformly when the options are shuffled. Any model should beat it. **Base + T** is the untuned LFM2.5-VL-1.6B, read the same way as the tuned model, with its label logits divided by one scalar temperature chosen to minimise NLL on the environment's development split. **Trained environments.** | environment | NLL | NLL, base + T | Brier | ECE | AUROC | acc | |---|---|---|---|---|---|---| | mcq | **0.427** | 0.527 | 0.218 | 0.012 | 0.877 | 0.847 | | ChaosNLI | **0.784** | 0.835 | 0.171 | 0.022 | 0.670 | 0.680 | | CivilComments | **0.323** | 0.629 | 0.044 | 0.064 | 0.922 | 0.933 | | STS-B | **1.033** | 1.749 | 0.302 | 0.045 | 0.621 | 0.572 | | Folktables | **0.448** | 0.612 | 0.296 | 0.013 | 0.761 | 0.778 | | CIFAR-10 (+C) | **0.253** | 0.526 | 0.111 | 0.006 | 0.947 | 0.922 | | Camelyon17 | **0.156** | 0.649 | 0.090 | 0.013 | 0.907 | 0.938 | > [!NOTE] > ChaosNLI, CivilComments and STS-B have soft labels from multiple annotations, so > top-label ECE does not fully measure calibration. On these, use NLL and KL divergence to > the annotator distribution. **Unseen tasks.** Not in training. | task | type | NLL | NLL, base + T | base rate | |---|---|---|---|---| | BoolQ | noul | 0.479 | **0.457** | 0.665 | | Yelp review stars | score | **1.110** | 1.226 | 1.609 | **Distribution shift.** CIFAR-10-C. | | clean | sev. 1 | 2 | 3 | 4 | 5 | |---|---|---|---|---|---|---| | accuracy | 0.982 | 0.952 | 0.935 | 0.917 | 0.892 | 0.848 | | mean confidence | 0.978 | 0.956 | 0.939 | 0.923 | 0.898 | 0.860 | **Other System One models.** NLL on the text environments, same test splits, same readout. The other alpha-sys-1 sizes on the table are their released seeds. `Qwen3.8-27B` is the open 27B generalist, read at its first answer token with reasoning off, plus a dev-fitted temperature. | environment | alpha-sys-1-450M | alpha-sys-1-1.6B (this) | alpha-sys-1-3B | Qwen3.8-27B + T | base rate | |---|---|---|---|---|---| | mcq | 0.727 | 0.427 | 0.289 | **0.159** | 1.439 | | ChaosNLI | 0.902 | 0.784 | 0.735 | **0.706** | 0.938 | | CivilComments | 0.324 | 0.323 | **0.319** | 0.473 | 0.425 | | STS-B | 1.107 | 1.033 | **0.961** | 1.347 | 1.727 | | Folktables | **0.436** | 0.448 | 0.441 | 0.472 | 0.683 | | BoolQ (unseen) | 0.661 | 0.479 | 0.405 | **0.316** | 0.665 | | Yelp review stars (unseen) | 1.467 | 1.110 | 0.973 | **0.858** | 1.609 | Per-hospital, per-state-year and per-identity-group tables, the three-seed gate tables and the full comparison against other System One models (hosted and open) are in the [repository](https://github.com/nullsilver-labs/alpha-sys-1) under `runs/`. ## Limitations > [!WARNING] > On a task that differs substantially from the training environments, do not assume this > model stays calibrated; measure it against the base model's calibration. In > leave-one-domain-out tests at 1.6B, a model tuned on the other environments beat the > untuned base with a transferred temperature on one held-out environment out of three, > and on the two unseen tasks above the trained-on-all checkpoints match the base model and do not beat it. > With a few hundred labelled examples from your own task, fit a temperature on them: > divide the label logits by one scalar chosen to minimise NLL on those examples > (`alpha_sys_1.fit_temperature`), then pass it as `SystemOne(..., temperature=T)`. - Knowledge depends on model size: on MCQ, the untuned 3B base outperforms this tuned model. - When the model does not know an answer, its distribution is close to uniform. - Under strong distribution shift, such as CIFAR-10-C at severity 5, confidence remains higher than accuracy. - Reversing the option order changes the top answer on 11% of MCQ items, mostly among low-confidence examples. - The answer space is capped at 26 options. - Fine-tuning used English data only and at most one image per question. ## Related work The interface follows TypeSafe's Jev (a hosted System One model, the `state` / `questions` request shape). Reading an answer distribution from the label-token logits of one forward pass is the readout of Kadavath et al. (2022, *Language Models (Mostly) Know What They Know*) and of the LLM-as-a-Verifier line of work, which scores rubric levels from the logits of letter tokens. That calibration improves with size, and that a temperature fitted on one domain transfers badly to another, is Jiang et al. (2021, *How Can We Know When Language Models Know?*). Training on a proper scoring rule against annotator distributions is why a fixed answer space and calibration are non-conflicting (Kalai and Vempala, 2024, *Calibrated Language Models Must Hallucinate*). Base models: Liquid AI's LFM2.5-VL. ## License This model is derived from LiquidAI/LFM2.5-VL-1.6B and is released under the [LFM Open License v1.0](LICENSE). ## Citation ```bibtex @misc{alphasys1, title = {alpha-sys-1: a small calibrated System One model}, author = {Nullsilver}, year = {2026}, url = {https://huggingface.co/collections/nullsilver/alpha-sys-1} } ```