sakshi-judge-indic

Groundedness/hallucination judge for banking-domain agent outputs in English, Hindi, and Hinglish — a QLoRA adapter on Qwen2.5-7B-Instruct. Given a CONTEXT and an ANSWER it emits one word — grounded or hallucinated — and the verdict token's probability is a calibrated confidence (ECE 0.013). Built by RotaVision for the Sakshi agent-governance platform; evaluated inside the customer environment, so no text leaves the deployment.

Results (held-out 600, disjoint pools and templates)

Metric	Result
Agreement	0.953 (gate ≥ 0.85)
ECE	0.013 (gate ≤ 0.10)
Per language	en 0.983 / hi 0.953 / hinglish 0.928

Per perturbation family: number_swap 1.00, negation 1.00, fabricated_fact 1.00, unsupported_claim 1.00 — and entity_swap 0.54, reported deliberately: the judge misses wrong-person/wrong-product substitutions about half the time. That is the v1 training priority. Score answers whose risk is entity identity accordingly.

Usage

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct", torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, "rotalabs/sakshi-judge-indic")
tok = AutoTokenizer.from_pretrained("rotalabs/sakshi-judge-indic")

SYSTEM = ("You are a groundedness judge for banking documents. Given a CONTEXT "
          "and an ANSWER, decide whether every claim in the answer is supported "
          "by the context. Reply with exactly one word: grounded or hallucinated.")
prompt = tok.apply_chat_template(
    [{"role": "system", "content": SYSTEM},
     {"role": "user", "content": "CONTEXT:\n...\n\nANSWER:\n...\n\nVerdict:"}],
    add_generation_prompt=True, return_tensors="pt").to(model.device)
logits = model(prompt).logits[0, -1]
g = tok("grounded", add_special_tokens=False)["input_ids"][0]
h = tok("hallucinated", add_special_tokens=False)["input_ids"][0]
probs = torch.softmax(torch.stack([logits[g], logits[h]]).float(), dim=0)
# probs[0] = P(grounded), probs[1] = P(hallucinated) — calibrated

Training data & honest scope

100% synthetic (context, answer, verdict) triples: banking passages rendered from structured facts; hallucinations produced by perturbing those facts (numbers, entities, negation) or injecting fabricated/unsupported claims — labels correct by construction, no distillation, no label noise. The gate set is construction-verified; human-verified evaluation is the v1 step. Trained on short factual banking passages — long multi-document contexts, tables, and reasoning-heavy answers are untested. A judge is a signal, not a verdict: route low-confidence or hallucinated outputs to human review.

Released by RotaVision under the rotalabs open-source commons, Apache-2.0. Generator, training and gated evaluation scripts live in the Sakshi repository.