JMasr/balidea-ood-domain-gate-es-gl-v2

balidea safeguard · ood_domain_gate · v2

Multilingual (ES/GL) safeguard classifier for clinical chatbot traffic — a full fine-tune of FacebookAI/xlm-roberta-base (no LoRA). Part of the v2 rebuild on a provenance-aware train/val/test methodology; supersedes the deprecated v1.

⚠️ Validation caveat. This model was trained with gold-in-train: ~176 rows of the production .ods acceptance set are in the training split, so the standard acceptance gate against the full .ods is not a clean check and was skipped for this upload. It is validated only on the held-out 20% gold-test slice below. The .ods is a curated 886-prompt set, not live production traffic — treat these numbers as a proxy and confirm on real traffic before relying on the model.

Model details

Field	Value
Base model	FacebookAI/xlm-roberta-base
Task / policy	ood_domain_gate / ood-gate-production-v2 (production-v2)
Languages	es, gl
Labels	0 = negative/pass, 1 = positive/flag
Decision threshold	0.8400
Temperature (Guo et al. 2017)	1.4534
Calibration slice	val (recall=0.9068825910931174, fp=0.0008539709649871904)
Dataset slug	v2-ood_domain_gate-seed20260526

Held-out gold-test (production .ods slice unseen in training)

n = 68 (26 positive, 42 negative).

Operating point	recall	FP	ROC-AUC	acceptance
shipped (threshold=0.840, val-calibrated)	0.3462	0.0	0.978	❌
gold-recalibrated (threshold=0.71)	0.9375	0.0	—	✅

ROC-AUC is threshold-free (model quality); recall/FP depend on the operating point. Confidence intervals and per-language slices are in the project's reports/v2/ood_domain_gate_lr_check_goldtrain.json.

Training

Known-good XLM-R recipe with early stopping (no epoch grid, no LoRA, no focal loss):

• epochs ceiling: 4 (early stopping, patience 2, metric eval_loss)
• learning rate: 3e-05
• weight decay: 0.01, warmup ratio: 0.06
• batch size: 16, max length: 256, seed: 20260526

Train split sources (n=13744, labels={'0': 9883, '1': 3861}):

source	rows
prod_medical_queries	2584
chitchat	2563
native_ood_es	2400
snomed_clinical	2030
prod_crisis_as_indomain	1648
native_ood_gl	1221
balidea_malign	948
ods	176
author_seeds	174

Usage

from safeguard.inference import SafeguardClassifier  # applies temperature + threshold

clf = SafeguardClassifier("ood_domain_gate-v2")  # or a local path / this repo id
print(clf.predict_detailed(["texto a clasificar"]))

Raw transformers (remember to apply temperature 1.453 and threshold 0.840):

from transformers import AutoModelForSequenceClassification, AutoTokenizer
model = AutoModelForSequenceClassification.from_pretrained("REPO_ID")
tokenizer = AutoTokenizer.from_pretrained("REPO_ID")