bojador-reporter-smollm3-3b

A reporter-only LoRA adapter for SmolLM3-3B. It makes the model's verbalized confidence discriminate its own errors, and it does so without changing the answers the model gives.

The trick is where the adapter is allowed to act. The frozen base model writes the answer. The adapter switches on only for a second turn that asks "what is your confidence?", and the training loss touches only those confidence tokens. Task capability therefore cannot degrade: nothing in the answer path was trained.

What it does

Asked, after answering, "Numa escala de 0 a 100, qual é a tua confiança NESSA resposta?", the model replies Confiança: N% with an N that ranks correct answers above incorrect ones. It was trained to distill a cross-fitted joint teacher (internal max-probability + the base model's own verbal confidence).

Results (see paper / repo for full tables)

• In-domain (synthetic ordering MCQ), confirmed & replicated: verbal error discrimination AUROC ≈0.87 (mean individual adapter), vs ≈0.75 for the best external use of the raw verbal signal and ≈0.73 for an S6-trained control. Replicated on an independent 400-item holdout (joint > control: ΔBrier +0.034, 95% CI [+0.010, +0.057]).
• Cross-domain (HellaSwag), descriptive: beats a frozen external calibration map zero-shot; the incremental advantage over the simpler control was not confirmed at the pre-registered gate.
• Not confirmed: Brier-calibration superiority over an external Platt calibrator (three measurements, all marginal). The confirmed value is discrimination, not absolute calibration.

Intended use

Research on small-model confidence, selective answering, and compute routing. Pair with the disagreement cascade (in the repo) for selective inference.

Limitations & scope

• One base model (SmolLM3-3B, 8-bit MLX), pinned revision 316f091e34982bc6eaf7f6cc1db82bdb77ac2103.
• Confirmatory results are within a synthetic generator; cross-domain increment did not confirm.
• Substantial train-seed heterogeneity (2/10 seed-pairs negative in-domain); multiple seeds are released — do not expect every seed to behave identically.
• Capability preservation holds for the reporter-only pipeline (adapter inactive while answering); not tested with the adapter active during answering.
• The confidence number is discriminative, not a calibrated probability out of the box; for absolute calibration, fit a destination calibrator.

Reproduce

Full pipeline, exact commands, and per-item artifacts with hashes: github.com/ajdramos/bojador → REPRODUCE.md. Training: src/tt4_data.py → src/train_lora.py (--mask-prompt); evaluation: src/tt4_eval.py.

License

Apache 2.0 (adapter and base model).