---
license: apache-2.0
base_model: mlx-community/SmolLM3-3B-8bit
tags:
  - confidence-estimation
  - calibration
  - selective-prediction
  - uncertainty
  - lora
  - mlx
  - small-models
language:
  - en
library_name: mlx
---

# bojador-reporter-smollm3-3b

A **reporter-only** LoRA adapter for
[SmolLM3-3B](https://huggingface.co/mlx-community/SmolLM3-3B-8bit). It makes the
model's verbalized confidence discriminate its own errors, and it does so without
changing the answers the model gives.

The trick is where the adapter is allowed to act. The frozen base model writes
the answer. The adapter switches on only for a second turn that asks "what is
your confidence?", and the training loss touches only those confidence tokens.
Task capability therefore cannot degrade: nothing in the answer path was trained.

## What it does

Asked, after answering, *"Numa escala de 0 a 100, qual é a tua confiança NESSA
resposta?"*, the model replies `Confiança: N%` with an N that ranks correct
answers above incorrect ones. It was trained to distill a cross-fitted joint
teacher (internal max-probability + the base model's own verbal confidence).

## Results (see paper / repo for full tables)

- **In-domain (synthetic ordering MCQ), confirmed & replicated:** verbal error
  discrimination AUROC ≈0.87 (mean individual adapter), vs ≈0.75 for the best
  *external* use of the raw verbal signal and ≈0.73 for an S6-trained control.
  Replicated on an independent 400-item holdout (joint > control: ΔBrier +0.034,
  95% CI [+0.010, +0.057]).
- **Cross-domain (HellaSwag), descriptive:** beats a frozen external calibration
  map zero-shot; the *incremental* advantage over the simpler control was **not
  confirmed** at the pre-registered gate.
- **Not confirmed:** Brier-calibration superiority over an external Platt
  calibrator (three measurements, all marginal). The confirmed value is
  **discrimination**, not absolute calibration.

## Intended use

Research on small-model confidence, selective answering, and compute routing.
Pair with the disagreement cascade (in the repo) for selective inference.

## Limitations & scope

- One base model (SmolLM3-3B, 8-bit MLX), pinned revision
  `316f091e34982bc6eaf7f6cc1db82bdb77ac2103`.
- Confirmatory results are **within a synthetic generator**; cross-domain
  *increment* did not confirm.
- Substantial **train-seed heterogeneity** (2/10 seed-pairs negative in-domain);
  multiple seeds are released — do not expect every seed to behave identically.
- Capability preservation holds **for the reporter-only pipeline** (adapter
  inactive while answering); not tested with the adapter active during answering.
- The confidence number is **discriminative, not a calibrated probability** out
  of the box; for absolute calibration, fit a destination calibrator.

## Reproduce

Full pipeline, exact commands, and per-item artifacts with hashes:
[github.com/ajdramos/bojador](https://github.com/ajdramos/bojador) →
`REPRODUCE.md`. Training: `src/tt4_data.py` → `src/train_lora.py`
(`--mask-prompt`); evaluation: `src/tt4_eval.py`.

## License

Apache 2.0 (adapter and base model).